[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:14:42 dist-test-slave-7z32 krb5kdc[28096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:14:42 dist-test-slave-7z32 krb5kdc[28096](info): set up 2 sockets
Aug 12 11:14:42 dist-test-slave-7z32 krb5kdc[28096](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 11:14:44.307835 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.444s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:14:44 dist-test-slave-7z32 krb5kdc[28096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:14:44.492676 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36139
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:44.668006 28115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:44.668335 28115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:44.668388 28115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:44.674405 28115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:14:44.674510 28115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:44.674531 28115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:44.674548 28115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:14:44.674566 28115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:14:44.681286 28115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:44.682690 28115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:44.684051 28115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:44.696594 28122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:44.696628 28123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:44.696628 28126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:44.696707 28115 server_base.cc:1034] running on GCE node
I20260812 11:14:44.697670 28115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:44.699227 28115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:44.700562 28115 hybrid_clock.cc:648] HybridClock initialized: now 1786533284700534 us; error 57 us; skew 500 ppm
Aug 12 11:14:44 dist-test-slave-7z32 krb5kdc[28096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:44.704793 28115 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:14:44.706090 28115 webserver.cc:460] Webserver started at http://127.27.109.62:36675/ using document root <none> and password file <none>
I20260812 11:14:44.706795 28115 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:44.706985 28115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:44.707266 28115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:44.709708 28115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "6c16b82a03dc42b2a151507d4b009a67"
format_stamp: "Formatted at 2026-08-12 11:14:44 on dist-test-slave-7z32"
server_key: "4341dffdb9fb5c7643e6a5b9bacd425f"
server_key_iv: "8ca5670170849ff93bac7696eb55a68d"
server_key_version: "encryptionkey@0"
I20260812 11:14:44.710361 28115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "6c16b82a03dc42b2a151507d4b009a67"
format_stamp: "Formatted at 2026-08-12 11:14:44 on dist-test-slave-7z32"
server_key: "4341dffdb9fb5c7643e6a5b9bacd425f"
server_key_iv: "8ca5670170849ff93bac7696eb55a68d"
server_key_version: "encryptionkey@0"
I20260812 11:14:44.715229 28115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:14:44.718675 28135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:44.720742 28115 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:14:44.720999 28115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "6c16b82a03dc42b2a151507d4b009a67"
format_stamp: "Formatted at 2026-08-12 11:14:44 on dist-test-slave-7z32"
server_key: "4341dffdb9fb5c7643e6a5b9bacd425f"
server_key_iv: "8ca5670170849ff93bac7696eb55a68d"
server_key_version: "encryptionkey@0"
I20260812 11:14:44.721175 28115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:44.740703 28115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:44.745493 28115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:44.745752 28115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:44.757105 28115 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:36139
I20260812 11:14:44.757265 28217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:36139 every 8 connection(s)
I20260812 11:14:44.758406 28115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:14:44.763532 28218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:44.765353 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28115
I20260812 11:14:44.765599 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:14:44.768194 28084 external_mini_cluster.cc:1428] Setting key 696bf5d793d1765c69cc8f9390e76875
I20260812 11:14:44.774433 28218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67: Bootstrap starting.
I20260812 11:14:44.777869 28218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:44.779006 28218 log.cc:824] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:44.782758 28218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67: No bootstrap required, opened a new log
I20260812 11:14:44.788482 28218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c16b82a03dc42b2a151507d4b009a67" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36139 } }
I20260812 11:14:44.788924 28218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:44.789003 28218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c16b82a03dc42b2a151507d4b009a67, State: Initialized, Role: FOLLOWER
I20260812 11:14:44.789716 28218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c16b82a03dc42b2a151507d4b009a67" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36139 } }
I20260812 11:14:44.789929 28218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:14:44.790035 28218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:14:44.790174 28218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:44.791916 28218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c16b82a03dc42b2a151507d4b009a67" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36139 } }
I20260812 11:14:44.792520 28218 leader_election.cc:302] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c16b82a03dc42b2a151507d4b009a67; no voters:
I20260812 11:14:44.793097 28218 leader_election.cc:288] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:14:44.793313 28225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:44.793751 28225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [term 1 LEADER]: Becoming Leader. State: Replica: 6c16b82a03dc42b2a151507d4b009a67, State: Running, Role: LEADER
I20260812 11:14:44.794253 28225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c16b82a03dc42b2a151507d4b009a67" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36139 } }
I20260812 11:14:44.795112 28218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 11:14:44 dist-test-slave-7z32 krb5kdc[28096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:44.796587 28230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c16b82a03dc42b2a151507d4b009a67. Latest consensus state: current_term: 1 leader_uuid: "6c16b82a03dc42b2a151507d4b009a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c16b82a03dc42b2a151507d4b009a67" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36139 } } }
I20260812 11:14:44.796758 28230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:44.797114 28226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c16b82a03dc42b2a151507d4b009a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c16b82a03dc42b2a151507d4b009a67" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36139 } } }
I20260812 11:14:44.797235 28226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67 [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:44.797806 28241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:14:44.801967 28241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:14:44.808152 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:44.773807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.27.109.62:36139)
0812 11:14:44.774612 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:44.774644 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:44.775756 (+ 1112us) server_negotiation.cc:407] Connection header received
0812 11:14:44.780364 (+ 4608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:44.780390 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:44.780997 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:44.781534 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:44.784786 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:44.786599 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:44.790184 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:44.790957 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:44.791140 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:44.798198 (+ 7058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:44.798208 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:44.798250 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:44.798303 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:44.801780 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:44.803127 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:44.803140 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:44.803149 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:44.803388 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:44.804031 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:44.804035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:44.804037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:44.804810 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:44.805115 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:44.806260 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0812 11:14:44.806759 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":254,"threads_started":1}
I20260812 11:14:44.812728 28241 catalog_manager.cc:1347] Generated new cluster ID: 6046b153f7dd4c2fbc57552e389dae52
I20260812 11:14:44.812863 28241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:14:44.827490 28241 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:14:44.828802 28241 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:14:44.835480 28241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c16b82a03dc42b2a151507d4b009a67: Generated new TSK 0
I20260812 11:14:44.836786 28241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:14:44.884032 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36139
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:45.053706 28262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:45.054113 28262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:45.054168 28262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:45.059464 28262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:45.059587 28262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:45.059727 28262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:14:45.067154 28262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36139
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:45.068470 28262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:45.069633 28262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:45.083256 28272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:45.083452 28271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:45.083595 28262 server_base.cc:1034] running on GCE node
W20260812 11:14:45.083256 28275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:45.084345 28262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:45.085103 28262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:45.086341 28262 hybrid_clock.cc:648] HybridClock initialized: now 1786533285086287 us; error 80 us; skew 500 ppm
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:45.090556 28262 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:14:45.092149 28262 webserver.cc:460] Webserver started at http://127.27.109.1:39383/ using document root <none> and password file <none>
I20260812 11:14:45.092914 28262 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:45.093030 28262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:45.093321 28262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:45.096583 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "cebbb77b98af4f1eb61951e36f235f89"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "e71c45c1f667b67a5947af60f499bdb0"
server_key_iv: "ff1d123aef7ec15c51cc38d8d5f0d68d"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.097420 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "cebbb77b98af4f1eb61951e36f235f89"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "e71c45c1f667b67a5947af60f499bdb0"
server_key_iv: "ff1d123aef7ec15c51cc38d8d5f0d68d"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.102694 28262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:14:45.106348 28285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.108870 28262 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:14:45.109284 28262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "cebbb77b98af4f1eb61951e36f235f89"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "e71c45c1f667b67a5947af60f499bdb0"
server_key_iv: "ff1d123aef7ec15c51cc38d8d5f0d68d"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.109731 28262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:45.127074 28262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:45.131067 28262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:45.131268 28262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:45.131906 28262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:45.132922 28262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:45.133014 28262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.133113 28262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:45.133157 28262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.145164 28262 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:46771
I20260812 11:14:45.145226 28464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:46771 every 8 connection(s)
I20260812 11:14:45.146335 28262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:14:45.148696 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28262
I20260812 11:14:45.148859 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:14:45.149227 28084 external_mini_cluster.cc:1428] Setting key cd366febdc4d9c50736d854adeb3979a
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:14:45.171188 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.149783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33943 (local address 127.27.109.62:36139)
0812 11:14:45.150018 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:45.150024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:45.151943 (+ 1919us) server_negotiation.cc:407] Connection header received
0812 11:14:45.153944 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:45.153949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:45.154037 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:45.154191 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:45.156893 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.157677 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.160506 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.160819 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.160971 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.166119 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:45.166123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:45.166126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:45.166161 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:45.168344 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.169143 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.169147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.169148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.169235 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.169642 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.169645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.169647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.169890 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:45.170016 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:45.170743 (+ 727us) server_negotiation.cc:299] Negotiation successful
0812 11:14:45.170984 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 11:14:45.172382 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.150275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36139 (local address 127.27.109.1:33943)
0812 11:14:45.151449 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:45.151659 (+ 210us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:45.153542 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:45.154509 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:45.154521 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:45.155408 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:45.156510 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:45.156546 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.157942 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.157950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:45.160166 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:45.160198 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.161134 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.161144 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:45.161540 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.162655 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:45.162692 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:45.165748 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:45.168564 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:45.168575 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:45.168608 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:45.169001 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:45.169340 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:45.169344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:45.169346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:45.169523 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:45.170016 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:45.170031 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:45.170450 (+ 419us) client_negotiation.cc:769] Sending connection context
0812 11:14:45.170679 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 11:14:45.171138 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":182,"threads_started":1}
I20260812 11:14:45.174899 28465 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36139
I20260812 11:14:45.175336 28465 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:45.176170 28465 heartbeater.cc:511] Master 127.27.109.62:36139 requested a full tablet report, sending...
I20260812 11:14:45.179738 28162 ts_manager.cc:194] Registered new tserver with Master: cebbb77b98af4f1eb61951e36f235f89 (127.27.109.1:46771)
I20260812 11:14:45.183221 28162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:33943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:14:45.205025 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36139
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:45.374104 28474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:45.374531 28474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:45.374586 28474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:45.379571 28474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:45.379699 28474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:45.379777 28474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:14:45.387516 28474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36139
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:45.389281 28474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:45.390970 28474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:45.406133 28488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:45.406133 28492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:45.406203 28474 server_base.cc:1034] running on GCE node
W20260812 11:14:45.406114 28487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:45.407567 28474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:45.408406 28474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:45.409821 28474 hybrid_clock.cc:648] HybridClock initialized: now 1786533285409786 us; error 60 us; skew 500 ppm
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:45.413403 28474 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:14:45.415031 28474 webserver.cc:460] Webserver started at http://127.27.109.2:45011/ using document root <none> and password file <none>
I20260812 11:14:45.415751 28474 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:45.415830 28474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:45.416035 28474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:45.418576 28474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "5b7f1e718e5e42b0b3f70d1daf81c282"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "1ee83412e90cc46c07f8ef246b666be7"
server_key_iv: "d25d931448a32cd4a097b346e50bb7dc"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.419351 28474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "5b7f1e718e5e42b0b3f70d1daf81c282"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "1ee83412e90cc46c07f8ef246b666be7"
server_key_iv: "d25d931448a32cd4a097b346e50bb7dc"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.424397 28474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:14:45.428309 28502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.430254 28474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:14:45.430469 28474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "5b7f1e718e5e42b0b3f70d1daf81c282"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "1ee83412e90cc46c07f8ef246b666be7"
server_key_iv: "d25d931448a32cd4a097b346e50bb7dc"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.430635 28474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:45.443121 28474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:45.447206 28474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:45.447497 28474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:45.448268 28474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:45.449301 28474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:45.449383 28474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.449472 28474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:45.449520 28474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.463294 28474 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:33861
I20260812 11:14:45.463332 28700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:33861 every 8 connection(s)
I20260812 11:14:45.464471 28474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:14:45.467971 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28474
I20260812 11:14:45.468106 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:14:45.468425 28084 external_mini_cluster.cc:1428] Setting key 34c21e38c326ee462dd2c50e414c41cd
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:14:45.486665 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.467513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:47217 (local address 127.27.109.62:36139)
0812 11:14:45.467774 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:45.467781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:45.468862 (+ 1081us) server_negotiation.cc:407] Connection header received
0812 11:14:45.470662 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:45.470666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:45.470734 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:45.470883 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:45.472403 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.473323 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.475517 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.475879 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.476007 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.480369 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:45.480374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:45.480376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:45.480417 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:45.482283 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.483898 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.483905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.483908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.484125 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.484826 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.484830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.484831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.485114 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:45.485225 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:45.486221 (+ 996us) server_negotiation.cc:299] Negotiation successful
0812 11:14:45.486456 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 11:14:45.487754 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.467953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36139 (local address 127.27.109.2:47217)
0812 11:14:45.468653 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:45.468717 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:45.470389 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:45.471128 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:45.471136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:45.471603 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:45.472197 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:45.472213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.473534 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.473541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:45.475318 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:45.475334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.476121 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.476129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:45.476501 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.477453 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:45.477486 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:45.480038 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:45.482508 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:45.482522 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:45.482585 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:45.483306 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:45.484330 (+ 1024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:45.484336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:45.484339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:45.484641 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:45.485256 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:45.485265 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:45.485680 (+ 415us) client_negotiation.cc:769] Sending connection context
0812 11:14:45.486096 (+ 416us) client_negotiation.cc:240] Negotiation successful
0812 11:14:45.486622 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":185,"threads_started":1}
I20260812 11:14:45.489315 28701 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36139
I20260812 11:14:45.489634 28701 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:45.490290 28701 heartbeater.cc:511] Master 127.27.109.62:36139 requested a full tablet report, sending...
I20260812 11:14:45.491694 28162 ts_manager.cc:194] Registered new tserver with Master: 5b7f1e718e5e42b0b3f70d1daf81c282 (127.27.109.2:33861)
I20260812 11:14:45.492702 28162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:47217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:14:45.512892 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36139
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:45.655838 28709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:45.656163 28709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:45.656212 28709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:45.660411 28709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:45.660501 28709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:45.660567 28709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:14:45.666378 28709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36139
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:45.667727 28709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:45.668855 28709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:45.681346 28718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:45.681357 28727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:45.681629 28721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:45.682160 28709 server_base.cc:1034] running on GCE node
I20260812 11:14:45.682930 28709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:45.683980 28709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:45.685281 28709 hybrid_clock.cc:648] HybridClock initialized: now 1786533285685252 us; error 49 us; skew 500 ppm
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:45.689637 28709 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:14:45.691107 28709 webserver.cc:460] Webserver started at http://127.27.109.3:45361/ using document root <none> and password file <none>
I20260812 11:14:45.691881 28709 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:45.691962 28709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:45.692222 28709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:45.695538 28709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "95123556542b48f4b5198148c7be8329"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "94c9921e5e3e4a53748b4405a99fa401"
server_key_iv: "0ded85acc15d0458b56f93134df1bef6"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.696485 28709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "95123556542b48f4b5198148c7be8329"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "94c9921e5e3e4a53748b4405a99fa401"
server_key_iv: "0ded85acc15d0458b56f93134df1bef6"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.702541 28709 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 11:14:45.707060 28739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.709447 28709 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:14:45.709738 28709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "95123556542b48f4b5198148c7be8329"
format_stamp: "Formatted at 2026-08-12 11:14:45 on dist-test-slave-7z32"
server_key: "94c9921e5e3e4a53748b4405a99fa401"
server_key_iv: "0ded85acc15d0458b56f93134df1bef6"
server_key_version: "encryptionkey@0"
I20260812 11:14:45.709960 28709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:45.732745 28709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:45.738045 28709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:45.738303 28709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:45.739048 28709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:45.740072 28709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:45.740127 28709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.740172 28709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:45.740242 28709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:45.752375 28709 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:40377
I20260812 11:14:45.752408 28926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:40377 every 8 connection(s)
I20260812 11:14:45.753523 28709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:14:45.755096 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28709
I20260812 11:14:45.755216 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:14:45.755602 28084 external_mini_cluster.cc:1428] Setting key bee3b834741460795ea16e2f83b58e2b
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:45.775162 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.756672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:58321 (local address 127.27.109.62:36139)
0812 11:14:45.756840 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:45.756844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:45.759089 (+ 2245us) server_negotiation.cc:407] Connection header received
0812 11:14:45.760151 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:45.760155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:45.760234 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:45.760366 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:45.761912 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.762997 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.764497 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.764958 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.765118 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.768966 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:45.768972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:45.768976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:45.769023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:45.771509 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.772268 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.772272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.772275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.772394 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.772788 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.772792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.772794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.773099 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:45.773227 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:45.774073 (+ 846us) server_negotiation.cc:299] Negotiation successful
0812 11:14:45.774289 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:14:45.775357 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.757390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36139 (local address 127.27.109.3:58321)
0812 11:14:45.758904 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:45.758964 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:45.759897 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:45.760664 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:45.760672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:45.761156 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:45.761736 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:45.761752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.763170 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.763173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:45.764320 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:45.764333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.765168 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.765171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:45.765378 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.766252 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:45.766274 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:45.768674 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:45.771701 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:45.771709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:45.771728 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:45.772140 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:45.772509 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:45.772512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:45.772513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:45.772678 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:45.773273 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:45.773280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:45.773686 (+ 406us) client_negotiation.cc:769] Sending connection context
0812 11:14:45.773904 (+ 218us) client_negotiation.cc:240] Negotiation successful
0812 11:14:45.774243 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":249,"threads_started":1}
I20260812 11:14:45.777002 28927 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36139
I20260812 11:14:45.777467 28927 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:45.778286 28927 heartbeater.cc:511] Master 127.27.109.62:36139 requested a full tablet report, sending...
I20260812 11:14:45.780062 28162 ts_manager.cc:194] Registered new tserver with Master: 95123556542b48f4b5198148c7be8329 (127.27.109.3:40377)
I20260812 11:14:45.780835 28162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:58321
I20260812 11:14:45.784577 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:14:45.826802 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.810479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.27.109.1:46771)
0812 11:14:45.811174 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:45.811182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:45.811217 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:14:45.811288 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:45.811302 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:45.811545 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:45.811718 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:45.812864 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.813877 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.815440 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.816255 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.816406 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.820386 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:45.820407 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:45.820438 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:45.820483 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:45.823174 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.824236 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.824245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.824247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.824407 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.824940 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.824944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.824945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.825321 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:45.825577 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:45.826095 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 11:14:45.826523 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":144,"threads_started":1}
W20260812 11:14:45.828115 28324 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49472
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:14:45 dist-test-slave-7z32 krb5kdc[28096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:14:45.869205 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:45.856619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.27.109.1:46771)
0812 11:14:45.856980 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:45.856987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:45.857186 (+ 199us) server_negotiation.cc:407] Connection header received
0812 11:14:45.857419 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:45.857427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:45.857487 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:45.857598 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:45.858656 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.859329 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.860259 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:45.860507 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:45.860624 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:45.864331 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:45.864336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:45.864339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:45.864385 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:45.866269 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.867046 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.867050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.867051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.867170 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:45.867586 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:45.867589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:45.867591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:45.867846 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:45.868022 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:45.868607 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 11:14:45.868917 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260812 11:14:45.871345 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28262
I20260812 11:14:45.907265 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28474
I20260812 11:14:45.956849 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28709
I20260812 11:14:45.974480 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28115
2026-08-12T11:14:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3160 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:14:46 dist-test-slave-7z32 krb5kdc[28956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:14:46 dist-test-slave-7z32 krb5kdc[28956](info): set up 2 sockets
Aug 12 11:14:46 dist-test-slave-7z32 krb5kdc[28956](info): commencing operation
krb5kdc: starting...
W20260812 11:14:47.363130 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.354s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:14:47 dist-test-slave-7z32 krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:14:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:14:47.531396 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43519
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:47.678642 28976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:47.679183 28976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:47.679240 28976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:47.683856 28976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:14:47.683961 28976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:47.683983 28976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:47.684003 28976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:14:47.684019 28976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:14:47.690623 28976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:47.692538 28976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:47.694099 28976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:47.706771 28989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:47.706741 28987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:47.706914 28976 server_base.cc:1034] running on GCE node
W20260812 11:14:47.706741 28986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:47.708350 28976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:47.709992 28976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:47.711347 28976 hybrid_clock.cc:648] HybridClock initialized: now 1786533287711316 us; error 58 us; skew 500 ppm
Aug 12 11:14:47 dist-test-slave-7z32 krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:47.715206 28976 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:14:47.716689 28976 webserver.cc:460] Webserver started at http://127.27.109.62:32885/ using document root <none> and password file <none>
I20260812 11:14:47.717526 28976 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:47.717644 28976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:47.717954 28976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:47.720992 28976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "bae2456d50cb4496ade4290f272e376f"
format_stamp: "Formatted at 2026-08-12 11:14:47 on dist-test-slave-7z32"
server_key: "73d3c8a27ba375e1b128747b86a60577"
server_key_iv: "f97f15717c1c4fd0d48747fef91c3824"
server_key_version: "encryptionkey@0"
I20260812 11:14:47.721817 28976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "bae2456d50cb4496ade4290f272e376f"
format_stamp: "Formatted at 2026-08-12 11:14:47 on dist-test-slave-7z32"
server_key: "73d3c8a27ba375e1b128747b86a60577"
server_key_iv: "f97f15717c1c4fd0d48747fef91c3824"
server_key_version: "encryptionkey@0"
I20260812 11:14:47.727525 28976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:14:47.731364 28999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:47.733285 28976 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:14:47.733539 28976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "bae2456d50cb4496ade4290f272e376f"
format_stamp: "Formatted at 2026-08-12 11:14:47 on dist-test-slave-7z32"
server_key: "73d3c8a27ba375e1b128747b86a60577"
server_key_iv: "f97f15717c1c4fd0d48747fef91c3824"
server_key_version: "encryptionkey@0"
I20260812 11:14:47.733785 28976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:47.758118 28976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:47.763496 28976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:47.763867 28976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:47.775017 28976 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:43519
I20260812 11:14:47.775048 29092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:43519 every 8 connection(s)
I20260812 11:14:47.776567 28976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:14:47.780488 29093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:47.783233 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28976
I20260812 11:14:47.783532 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:14:47.783912 28084 external_mini_cluster.cc:1428] Setting key 59f9e28851895fcb9b025e51ac8c2f5d
I20260812 11:14:47.789441 29093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f: Bootstrap starting.
I20260812 11:14:47.792820 29093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:47.793830 29093 log.cc:824] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:47.796720 29093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f: No bootstrap required, opened a new log
Aug 12 11:14:47 dist-test-slave-7z32 krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:47.801015 29093 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2456d50cb4496ade4290f272e376f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43519 } }
I20260812 11:14:47.801374 29093 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:47.801429 29093 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae2456d50cb4496ade4290f272e376f, State: Initialized, Role: FOLLOWER
I20260812 11:14:47.801986 29093 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2456d50cb4496ade4290f272e376f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43519 } }
I20260812 11:14:47.802162 29093 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:14:47.802245 29093 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:14:47.802397 29093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:47.803629 29093 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2456d50cb4496ade4290f272e376f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43519 } }
I20260812 11:14:47.804068 29093 leader_election.cc:302] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae2456d50cb4496ade4290f272e376f; no voters:
I20260812 11:14:47.804504 29093 leader_election.cc:288] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:14:47.804685 29100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:47.804952 29100 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [term 1 LEADER]: Becoming Leader. State: Replica: bae2456d50cb4496ade4290f272e376f, State: Running, Role: LEADER
I20260812 11:14:47.805367 29100 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2456d50cb4496ade4290f272e376f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43519 } }
I20260812 11:14:47.806200 29093 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:14:47.806506 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:47.786065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.27.109.62:43519)
0812 11:14:47.786746 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:47.786772 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:47.786979 (+ 207us) server_negotiation.cc:407] Connection header received
0812 11:14:47.788031 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:47.788065 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:47.788626 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:47.789103 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:47.790188 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:47.792018 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:47.793233 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:47.793964 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:47.794257 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:47.797732 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:47.797741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:47.797767 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:47.797815 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:47.801294 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:47.802201 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:47.802236 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:47.802248 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:47.802524 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:47.802983 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:47.802988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:47.802990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:47.803675 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:47.803994 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:47.804415 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 11:14:47.805136 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":157,"threads_started":1}
I20260812 11:14:47.807907 29101 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae2456d50cb4496ade4290f272e376f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2456d50cb4496ade4290f272e376f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43519 } } }
I20260812 11:14:47.808076 29101 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:47.807848 29102 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae2456d50cb4496ade4290f272e376f. Latest consensus state: current_term: 1 leader_uuid: "bae2456d50cb4496ade4290f272e376f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2456d50cb4496ade4290f272e376f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43519 } } }
I20260812 11:14:47.808341 29102 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:47.809942 29110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:14:47.814190 29110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:14:47.822119 29110 catalog_manager.cc:1347] Generated new cluster ID: 1a295104be3e4668883dd179482264f7
I20260812 11:14:47.822232 29110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:14:47.857909 29110 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:14:47.859440 29110 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:14:47.867175 29110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae2456d50cb4496ade4290f272e376f: Generated new TSK 0
I20260812 11:14:47.868149 29110 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:14:47.915942 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43519
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:48.074015 29129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:48.074446 29129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:48.074502 29129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:48.079445 29129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:48.079553 29129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:48.079628 29129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:14:48.086134 29129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:48.087936 29129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:48.089329 29129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:14:48.103386 29129 server_base.cc:1034] running on GCE node
W20260812 11:14:48.103291 29136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:48.103276 29135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:48.103276 29138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:48.104507 29129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:48.105578 29129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:48.106912 29129 hybrid_clock.cc:648] HybridClock initialized: now 1786533288106887 us; error 67 us; skew 500 ppm
Aug 12 11:14:48 dist-test-slave-7z32 krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:48.117151 29129 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:14:48.118688 29129 webserver.cc:460] Webserver started at http://127.27.109.1:35485/ using document root <none> and password file <none>
I20260812 11:14:48.119710 29129 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:48.119807 29129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:48.120113 29129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:48.123031 29129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "28543489e8ad4a34afdec417f93cfdd5"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "ef27ca50638223ee4bff9e6aceca57ad"
server_key_iv: "f4e8c514150395c6e63e99a5bdd95218"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.123750 29129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "28543489e8ad4a34afdec417f93cfdd5"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "ef27ca50638223ee4bff9e6aceca57ad"
server_key_iv: "f4e8c514150395c6e63e99a5bdd95218"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.128840 29129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:14:48.132333 29153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.134275 29129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:14:48.134491 29129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "28543489e8ad4a34afdec417f93cfdd5"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "ef27ca50638223ee4bff9e6aceca57ad"
server_key_iv: "f4e8c514150395c6e63e99a5bdd95218"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.134649 29129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:48.150314 29129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:48.155411 29129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:48.155669 29129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:48.156533 29129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:48.157871 29129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:48.157936 29129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.157989 29129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:48.158006 29129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.173635 29129 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:44807
I20260812 11:14:48.173666 29319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:44807 every 8 connection(s)
I20260812 11:14:48.175230 29129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:14:48.178819 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29129
I20260812 11:14:48.179042 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:14:48.179426 28084 external_mini_cluster.cc:1428] Setting key c50de07a49a809c461d5b440e4e07d87
Aug 12 11:14:48 dist-test-slave-7z32 krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:14:48.197719 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.178016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:41389 (local address 127.27.109.62:43519)
0812 11:14:48.178302 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:48.178308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:48.179984 (+ 1676us) server_negotiation.cc:407] Connection header received
0812 11:14:48.181581 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:48.181585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:48.181882 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:48.182090 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:48.184424 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.185236 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.187463 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.187807 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.187962 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.192235 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:48.192240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:48.192242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:48.192284 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:48.194347 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.195402 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.195406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.195408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.195514 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.195995 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.195998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.196001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.196357 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:48.196482 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:48.197293 (+ 811us) server_negotiation.cc:299] Negotiation successful
0812 11:14:48.197491 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 11:14:48.199051 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.178686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43519 (local address 127.27.109.1:41389)
0812 11:14:48.179659 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:48.179772 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:48.181189 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:48.182472 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:48.182483 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:48.183213 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:48.184077 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.184104 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.185468 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.185475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.187211 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.187232 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.188065 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.188072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.188380 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.189229 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:48.189259 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:48.191875 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:48.194546 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:48.194557 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:48.194590 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:48.195246 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:48.195644 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:48.195648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:48.195650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:48.195864 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:48.196481 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:48.196492 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:48.196950 (+ 458us) client_negotiation.cc:769] Sending connection context
0812 11:14:48.197222 (+ 272us) client_negotiation.cc:240] Negotiation successful
0812 11:14:48.197747 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":242,"threads_started":1}
I20260812 11:14:48.201457 29320 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43519
I20260812 11:14:48.201828 29320 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:48.202564 29320 heartbeater.cc:511] Master 127.27.109.62:43519 requested a full tablet report, sending...
I20260812 11:14:48.205189 29024 ts_manager.cc:194] Registered new tserver with Master: 28543489e8ad4a34afdec417f93cfdd5 (127.27.109.1:44807)
I20260812 11:14:48.207436 29024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:41389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:14:48.224572 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43519
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:48.417682 29328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:48.418080 29328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:48.418135 29328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:48.423089 29328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:48.423189 29328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:48.423297 29328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:14:48.429955 29328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:48.431723 29328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:48.433279 29328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:48.447256 29336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:48.447243 29338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:48.447270 29335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:48.447604 29328 server_base.cc:1034] running on GCE node
I20260812 11:14:48.448383 29328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:48.449363 29328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:48.450608 29328 hybrid_clock.cc:648] HybridClock initialized: now 1786533288450561 us; error 66 us; skew 500 ppm
Aug 12 11:14:48 dist-test-slave-7z32 krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:48.455034 29328 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:14:48.456416 29328 webserver.cc:460] Webserver started at http://127.27.109.2:45975/ using document root <none> and password file <none>
I20260812 11:14:48.457214 29328 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:48.457329 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:48.457630 29328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:48.460495 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "d0b056aa7d5f4517a28bac0986bbd94d"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "fef5e1329963fc95a8c1c0adaafdd62a"
server_key_iv: "0215abe1f7c847b5b7e133b384fd3d10"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.461238 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "d0b056aa7d5f4517a28bac0986bbd94d"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "fef5e1329963fc95a8c1c0adaafdd62a"
server_key_iv: "0215abe1f7c847b5b7e133b384fd3d10"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.466779 29328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:14:48.470921 29351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.473485 29328 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260812 11:14:48.473771 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "d0b056aa7d5f4517a28bac0986bbd94d"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "fef5e1329963fc95a8c1c0adaafdd62a"
server_key_iv: "0215abe1f7c847b5b7e133b384fd3d10"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.473976 29328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:48.497969 29328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:48.502542 29328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:48.502883 29328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:48.503710 29328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:48.504896 29328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:48.504957 29328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.505010 29328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:48.505026 29328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.518497 29328 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:41431
I20260812 11:14:48.518519 29518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:41431 every 8 connection(s)
I20260812 11:14:48.519894 29328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:14:48.524533 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29328
I20260812 11:14:48.524663 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:14:48.524993 28084 external_mini_cluster.cc:1428] Setting key d4dfcb18b349d6bf82ebea8780d7fc00
Aug 12 11:14:48 dist-test-slave-7z32 krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:14:48.539511 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.522404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:51053 (local address 127.27.109.62:43519)
0812 11:14:48.522660 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:48.522667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:48.524386 (+ 1719us) server_negotiation.cc:407] Connection header received
0812 11:14:48.525635 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:48.525638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:48.525714 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:48.525835 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:48.527306 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.527880 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.529146 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.529422 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.529561 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.533490 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:48.533497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:48.533501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:48.533548 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:48.535661 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.536569 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.536576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.536578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.536711 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.537241 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.537245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.537247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.537632 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:48.537773 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:48.539132 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0812 11:14:48.539279 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 11:14:48.541230 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.523114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43519 (local address 127.27.109.2:51053)
0812 11:14:48.524155 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:48.524215 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:48.525334 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:48.526041 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:48.526050 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:48.526485 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:48.527128 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.527143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.528034 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.528037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.529001 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.529012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.529710 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.529717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.529980 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.530876 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:48.530905 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:48.533187 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:48.535886 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:48.535895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:48.535919 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:48.536374 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:48.536866 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:48.536869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:48.536871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:48.537082 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:48.537847 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:48.537856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:48.538474 (+ 618us) client_negotiation.cc:769] Sending connection context
0812 11:14:48.539042 (+ 568us) client_negotiation.cc:240] Negotiation successful
0812 11:14:48.539760 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":182,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:14:48.544401 29519 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43519
I20260812 11:14:48.544888 29519 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:48.545830 29519 heartbeater.cc:511] Master 127.27.109.62:43519 requested a full tablet report, sending...
I20260812 11:14:48.548076 29024 ts_manager.cc:194] Registered new tserver with Master: d0b056aa7d5f4517a28bac0986bbd94d (127.27.109.2:41431)
I20260812 11:14:48.549353 29024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:51053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:14:48.570250 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43519
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:48.729996 29530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:48.730413 29530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:48.730468 29530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:48.735687 29530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:48.735822 29530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:48.735904 29530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:14:48.744244 29530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:48.746361 29530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:48.748054 29530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:48.763183 29542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:48.763175 29545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:48.763391 29530 server_base.cc:1034] running on GCE node
W20260812 11:14:48.763175 29548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:48.764279 29530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:48.765805 29530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:48.767083 29530 hybrid_clock.cc:648] HybridClock initialized: now 1786533288767046 us; error 52 us; skew 500 ppm
Aug 12 11:14:48 dist-test-slave-7z32 krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:48.771384 29530 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:14:48.773018 29530 webserver.cc:460] Webserver started at http://127.27.109.3:41907/ using document root <none> and password file <none>
I20260812 11:14:48.773877 29530 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:48.773967 29530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:48.774277 29530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:48.777326 29530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "5043c72460014a9bbdd7ef6e6360ad82"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "25214427f43a322eafe5276f7e573648"
server_key_iv: "ebafa40f09edc24113ba1039599daca3"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.778167 29530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "5043c72460014a9bbdd7ef6e6360ad82"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "25214427f43a322eafe5276f7e573648"
server_key_iv: "ebafa40f09edc24113ba1039599daca3"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.785207 29530 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 11:14:48.789722 29559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.792546 29530 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:14:48.792804 29530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "5043c72460014a9bbdd7ef6e6360ad82"
format_stamp: "Formatted at 2026-08-12 11:14:48 on dist-test-slave-7z32"
server_key: "25214427f43a322eafe5276f7e573648"
server_key_iv: "ebafa40f09edc24113ba1039599daca3"
server_key_version: "encryptionkey@0"
I20260812 11:14:48.793004 29530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:48.810964 29530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:48.816090 29530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:48.816412 29530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:48.817307 29530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:48.818574 29530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:48.818665 29530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.818753 29530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:48.818804 29530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:48.833137 29530 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:41383
I20260812 11:14:48.833354 29744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:41383 every 8 connection(s)
I20260812 11:14:48.834509 29530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:14:48.843850 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29530
I20260812 11:14:48.844024 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:14:48.844389 28084 external_mini_cluster.cc:1428] Setting key 0f0b6e0dde10180485cf0d45547d1c62
Aug 12 11:14:48 dist-test-slave-7z32 krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:48.856446 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.837117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:54807 (local address 127.27.109.62:43519)
0812 11:14:48.837289 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:48.837293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:48.838963 (+ 1670us) server_negotiation.cc:407] Connection header received
0812 11:14:48.840739 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:48.840743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:48.840905 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:48.841074 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:48.842928 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.843674 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.845815 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.846093 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.846252 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.849880 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:48.849885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:48.849889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:48.849920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:48.852399 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.853339 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.853343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.853345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.853515 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.854500 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.854503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.854505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.855014 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:48.855149 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:48.856036 (+ 887us) server_negotiation.cc:299] Negotiation successful
0812 11:14:48.856261 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:14:48.857650 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.837545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43519 (local address 127.27.109.3:54807)
0812 11:14:48.838578 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:48.838647 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:48.840272 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:48.841385 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:48.841394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:48.841920 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:48.842643 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.842669 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.844003 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.844011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.845575 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.845599 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.846353 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.846362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.846617 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.847508 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:48.847558 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:48.849622 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:48.852631 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:48.852640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:48.852664 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:48.853124 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:48.853946 (+ 822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:48.853950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:48.853952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:48.854146 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:48.855209 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:48.855218 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:48.855628 (+ 410us) client_negotiation.cc:769] Sending connection context
0812 11:14:48.855879 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 11:14:48.856345 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":162,"threads_started":1}
I20260812 11:14:48.859845 29745 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43519
I20260812 11:14:48.860275 29745 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:48.861157 29745 heartbeater.cc:511] Master 127.27.109.62:43519 requested a full tablet report, sending...
I20260812 11:14:48.862607 29021 ts_manager.cc:194] Registered new tserver with Master: 5043c72460014a9bbdd7ef6e6360ad82 (127.27.109.3:41383)
I20260812 11:14:48.863632 29021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:54807
I20260812 11:14:48.875012 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:14:48.890524 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.880350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.27.109.62:43519)
0812 11:14:48.880540 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:48.880545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:48.880570 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:14:48.880720 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:48.880724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:48.880795 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:48.880887 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:48.881899 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.882546 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.883754 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.884027 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.884141 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.886242 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:48.886247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:48.886249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:48.886291 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:48.888314 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.888961 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.888964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.888966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.889080 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:48.889383 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:48.889386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:48.889387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:48.889708 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:48.889848 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:48.890205 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 11:14:48.890361 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:14:48.897265 29021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:48.901633 29021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:48.924504 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.915704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41383 (local address 127.0.0.1:52622)
0812 11:14:48.917464 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:48.917537 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:48.917772 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:48.918580 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:48.918587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:48.918616 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:48.918800 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.918808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.920496 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.920500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.921971 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.921984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.923765 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.923770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.923929 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.923977 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 11:14:48.924113 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 11:14:48.924279 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1520,"mutex_wait_us":84,"thread_start_us":176,"threads_started":1}
I20260812 11:14:48.924516 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.916774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44807 (local address 127.0.0.1:36760)
0812 11:14:48.918065 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:48.918076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:48.918226 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:48.918902 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:48.918906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:48.918930 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:48.919076 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.919085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.920444 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.920448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.922324 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.922342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.924168 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.924172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.924255 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.924276 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:14:48.924348 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 11:14:48.924416 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"mutex_wait_us":343,"thread_start_us":147,"threads_started":1}
I20260812 11:14:48.925377 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.916949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.27.109.1:44807)
0812 11:14:48.917493 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:48.917502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:48.918177 (+ 675us) server_negotiation.cc:407] Connection header received
0812 11:14:48.918450 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:48.918459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:48.918644 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:48.918933 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:48.919265 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.920305 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.922983 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.923762 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.923908 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.924100 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:48.924736 (+ 636us) server_negotiation.cc:299] Negotiation successful
0812 11:14:48.924947 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260812 11:14:48.925383 29775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.917064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.27.109.2:41431)
0812 11:14:48.918606 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:48.918620 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:48.918642 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:14:48.918723 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:48.918729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:48.919016 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:48.919195 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:48.919419 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.920575 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.922550 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.924190 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.924319 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.924474 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:48.925012 (+ 538us) server_negotiation.cc:299] Negotiation successful
0812 11:14:48.925157 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1383,"thread_start_us":213,"threads_started":1}
I20260812 11:14:48.925637 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.915821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.27.109.3:41383)
0812 11:14:48.916410 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:48.916428 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:48.917675 (+ 1247us) server_negotiation.cc:407] Connection header received
0812 11:14:48.918034 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:48.918041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:48.918350 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:48.918571 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:48.919011 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.920291 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.922235 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.923558 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.923730 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.923955 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:48.925287 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0812 11:14:48.925467 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":224,"threads_started":1}
I20260812 11:14:48.925952 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:48.915826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:41431 (local address 127.0.0.1:34440)
0812 11:14:48.917588 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:48.917605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:48.917772 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:48.919184 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:48.919189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:48.919211 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:48.919301 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.919310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.920731 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.920734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.922324 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:48.922342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:48.924542 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:48.924546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:48.924697 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:48.924724 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:14:48.924802 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 11:14:48.924904 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"mutex_wait_us":65,"thread_start_us":229,"threads_started":1}
I20260812 11:14:48.929647 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 9947ec9d50fb41ed9a169cd14d1d66db (DEFAULT_TABLE table=test-table [id=92d30d2df8354e76a180b91d955a15ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:48.929880 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 9947ec9d50fb41ed9a169cd14d1d66db (DEFAULT_TABLE table=test-table [id=92d30d2df8354e76a180b91d955a15ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:48.929883 29635 tablet_service.cc:1467] Processing CreateTablet for tablet 9947ec9d50fb41ed9a169cd14d1d66db (DEFAULT_TABLE table=test-table [id=92d30d2df8354e76a180b91d955a15ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:48.931036 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9947ec9d50fb41ed9a169cd14d1d66db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:48.931408 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9947ec9d50fb41ed9a169cd14d1d66db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:48.931748 29635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9947ec9d50fb41ed9a169cd14d1d66db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:48.938776 29776 tablet_bootstrap.cc:492] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d: Bootstrap starting.
I20260812 11:14:48.942080 29778 tablet_bootstrap.cc:492] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5: Bootstrap starting.
I20260812 11:14:48.943190 29776 tablet_bootstrap.cc:654] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:48.944031 29779 tablet_bootstrap.cc:492] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82: Bootstrap starting.
I20260812 11:14:48.944828 29776 log.cc:824] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:48.945595 29778 tablet_bootstrap.cc:654] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:48.946532 29778 log.cc:824] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:48.947646 29779 tablet_bootstrap.cc:654] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:48.947821 29776 tablet_bootstrap.cc:492] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d: No bootstrap required, opened a new log
I20260812 11:14:48.948112 29776 ts_tablet_manager.cc:1397] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260812 11:14:48.949393 29778 tablet_bootstrap.cc:492] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5: No bootstrap required, opened a new log
I20260812 11:14:48.949388 29779 log.cc:824] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:48.949609 29778 ts_tablet_manager.cc:1397] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 11:14:48.952596 29776 raft_consensus.cc:348] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:48.952945 29776 raft_consensus.cc:374] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:48.953001 29776 raft_consensus.cc:729] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b056aa7d5f4517a28bac0986bbd94d, State: Initialized, Role: FOLLOWER
I20260812 11:14:48.953549 29776 consensus_queue.cc:260] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:48.954358 29778 raft_consensus.cc:348] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:48.954635 29778 raft_consensus.cc:374] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:48.954644 29519 heartbeater.cc:503] Master 127.27.109.62:43519 was elected leader, sending a full tablet report...
I20260812 11:14:48.954713 29778 raft_consensus.cc:729] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28543489e8ad4a34afdec417f93cfdd5, State: Initialized, Role: FOLLOWER
I20260812 11:14:48.954816 29776 ts_tablet_manager.cc:1428] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260812 11:14:48.955302 29778 consensus_queue.cc:260] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:48.956220 29320 heartbeater.cc:503] Master 127.27.109.62:43519 was elected leader, sending a full tablet report...
I20260812 11:14:48.956449 29778 ts_tablet_manager.cc:1428] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 11:14:48.958287 29779 tablet_bootstrap.cc:492] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82: No bootstrap required, opened a new log
I20260812 11:14:48.958704 29779 ts_tablet_manager.cc:1397] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260812 11:14:48.964166 29779 raft_consensus.cc:348] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:48.964588 29779 raft_consensus.cc:374] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:48.964629 29779 raft_consensus.cc:729] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5043c72460014a9bbdd7ef6e6360ad82, State: Initialized, Role: FOLLOWER
I20260812 11:14:48.965286 29779 consensus_queue.cc:260] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:48.966367 29745 heartbeater.cc:503] Master 127.27.109.62:43519 was elected leader, sending a full tablet report...
I20260812 11:14:48.967068 29779 ts_tablet_manager.cc:1428] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82: Time spent starting tablet: real 0.008s user 0.009s sys 0.001s
W20260812 11:14:49.021397 29520 tablet.cc:2369] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:14:49.085963 29746 tablet.cc:2369] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:14:49.177594 29322 tablet.cc:2369] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:14:49.196198 29790 raft_consensus.cc:482] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:49.196512 29790 raft_consensus.cc:504] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:49.198189 29790 leader_election.cc:288] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5043c72460014a9bbdd7ef6e6360ad82 (127.27.109.3:41383), d0b056aa7d5f4517a28bac0986bbd94d (127.27.109.2:41431)
I20260812 11:14:49.203603 29775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:49.199066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:42069 (local address 127.27.109.2:41431)
0812 11:14:49.199309 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:49.199317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:49.199665 (+ 348us) server_negotiation.cc:407] Connection header received
0812 11:14:49.199841 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:49.199845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:49.199910 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:49.200021 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:49.200335 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.200915 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.202349 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.202806 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.203042 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:49.203113 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:49.203314 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 11:14:49.203411 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:14:49.203685 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:49.198882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:41431 (local address 127.27.109.1:42069)
0812 11:14:49.199514 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:49.199537 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:49.199722 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:49.200038 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:49.200041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:49.200062 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:49.200203 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:49.200209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.201048 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.201053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:49.202181 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:49.202193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.203042 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.203047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:49.203158 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:49.203192 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:14:49.203293 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 11:14:49.203404 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":44,"thread_start_us":338,"threads_started":1}
I20260812 11:14:49.203863 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:49.199005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:53615 (local address 127.27.109.3:41383)
0812 11:14:49.199185 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:49.199191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:49.199744 (+ 553us) server_negotiation.cc:407] Connection header received
0812 11:14:49.199958 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:49.199961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:49.200020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:49.200122 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:49.200432 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.201009 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.202364 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.202899 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.203069 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:49.203163 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:49.203364 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 11:14:49.203458 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:14:49.203876 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:49.198882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41383 (local address 127.27.109.1:53615)
0812 11:14:49.199625 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:49.199646 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:49.199794 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:49.200138 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:49.200141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:49.200159 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:49.200309 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:49.200315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.201146 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.201152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:49.202187 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:49.202198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.203069 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.203073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:49.203200 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:49.203232 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:14:49.203340 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 11:14:49.203455 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":127,"thread_start_us":322,"threads_started":1}
I20260812 11:14:49.204591 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9947ec9d50fb41ed9a169cd14d1d66db" candidate_uuid: "28543489e8ad4a34afdec417f93cfdd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" is_pre_election: true
I20260812 11:14:49.204890 29672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9947ec9d50fb41ed9a169cd14d1d66db" candidate_uuid: "28543489e8ad4a34afdec417f93cfdd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5043c72460014a9bbdd7ef6e6360ad82" is_pre_election: true
I20260812 11:14:49.205030 29451 raft_consensus.cc:2454] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28543489e8ad4a34afdec417f93cfdd5 in term 0.
I20260812 11:14:49.205204 29672 raft_consensus.cc:2454] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28543489e8ad4a34afdec417f93cfdd5 in term 0.
I20260812 11:14:49.205755 29155 leader_election.cc:302] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28543489e8ad4a34afdec417f93cfdd5, d0b056aa7d5f4517a28bac0986bbd94d; no voters:
I20260812 11:14:49.206410 29790 raft_consensus.cc:2781] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:49.206688 29790 raft_consensus.cc:482] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:49.206797 29790 raft_consensus.cc:3037] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:49.209055 29790 raft_consensus.cc:504] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:49.209697 29790 leader_election.cc:288] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [CANDIDATE]: Term 1 election: Requested vote from peers 5043c72460014a9bbdd7ef6e6360ad82 (127.27.109.3:41383), d0b056aa7d5f4517a28bac0986bbd94d (127.27.109.2:41431)
I20260812 11:14:49.210801 29672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9947ec9d50fb41ed9a169cd14d1d66db" candidate_uuid: "28543489e8ad4a34afdec417f93cfdd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5043c72460014a9bbdd7ef6e6360ad82"
I20260812 11:14:49.210788 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9947ec9d50fb41ed9a169cd14d1d66db" candidate_uuid: "28543489e8ad4a34afdec417f93cfdd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b056aa7d5f4517a28bac0986bbd94d"
I20260812 11:14:49.211062 29451 raft_consensus.cc:3037] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:49.211062 29672 raft_consensus.cc:3037] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:49.212723 29451 raft_consensus.cc:2454] T 9947ec9d50fb41ed9a169cd14d1d66db P d0b056aa7d5f4517a28bac0986bbd94d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28543489e8ad4a34afdec417f93cfdd5 in term 1.
I20260812 11:14:49.212723 29672 raft_consensus.cc:2454] T 9947ec9d50fb41ed9a169cd14d1d66db P 5043c72460014a9bbdd7ef6e6360ad82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28543489e8ad4a34afdec417f93cfdd5 in term 1.
I20260812 11:14:49.213294 29154 leader_election.cc:302] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28543489e8ad4a34afdec417f93cfdd5, 5043c72460014a9bbdd7ef6e6360ad82; no voters:
I20260812 11:14:49.213730 29790 raft_consensus.cc:2781] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:49.214111 29790 raft_consensus.cc:686] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [term 1 LEADER]: Becoming Leader. State: Replica: 28543489e8ad4a34afdec417f93cfdd5, State: Running, Role: LEADER
I20260812 11:14:49.214658 29790 consensus_queue.cc:237] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } }
I20260812 11:14:49.220263 29019 catalog_manager.cc:5495] T 9947ec9d50fb41ed9a169cd14d1d66db P 28543489e8ad4a34afdec417f93cfdd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28543489e8ad4a34afdec417f93cfdd5 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "28543489e8ad4a34afdec417f93cfdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5043c72460014a9bbdd7ef6e6360ad82" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b056aa7d5f4517a28bac0986bbd94d" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28543489e8ad4a34afdec417f93cfdd5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44807 } health_report { overall_health: HEALTHY } } }
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:14:49.310686 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:49.295739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.27.109.1:44807)
0812 11:14:49.295994 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:49.295998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:49.296151 (+ 153us) server_negotiation.cc:407] Connection header received
0812 11:14:49.296395 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:49.296399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:49.296466 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:49.296579 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:49.298075 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.299005 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.300593 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.300893 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.301051 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:49.304981 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:49.305001 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:49.305021 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:49.305062 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:49.307433 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:49.308405 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:49.308412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:49.308414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:49.308583 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:49.309171 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:49.309174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:49.309176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:49.309583 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:49.309779 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:49.310177 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 11:14:49.310450 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533289, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533289, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:14:49.357823 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:49.342182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.27.109.1:44807)
0812 11:14:49.342399 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:49.342405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:49.342756 (+ 351us) server_negotiation.cc:407] Connection header received
0812 11:14:49.343331 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:49.343335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:49.343412 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:49.343513 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:49.345056 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.346179 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.347809 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.348240 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.348481 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:49.352405 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:49.352411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:49.352414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:49.352468 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:49.354477 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:49.355615 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:49.355620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:49.355621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:49.355761 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:49.356353 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:49.356357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:49.356359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:49.356710 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:49.356868 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:49.357400 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 11:14:49.357631 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:14:49.402398 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:49.387517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.27.109.1:44807)
0812 11:14:49.387822 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:49.387829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:49.388173 (+ 344us) server_negotiation.cc:407] Connection header received
0812 11:14:49.388428 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:49.388432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:49.388490 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:49.388674 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:49.390680 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.392190 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.393294 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:49.393551 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:49.393681 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:49.397246 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:49.397251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:49.397254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:49.397294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:49.399278 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:49.399878 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:49.399885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:49.399887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:49.400088 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:49.400466 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:49.400470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:49.400472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:49.400692 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:49.401018 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:49.401875 (+ 857us) server_negotiation.cc:299] Negotiation successful
0812 11:14:49.402189 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260812 11:14:49.404877 29194 tablet_service.cc:2908] Rejecting scan request for tablet 9947ec9d50fb41ed9a169cd14d1d66db: Uninitialized: clean time has not yet been initialized
I20260812 11:14:49.408957 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29129
I20260812 11:14:49.427129 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29328
I20260812 11:14:49.489065 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29530
I20260812 11:14:49.506194 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28976
2026-08-12T11:14:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3534 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[29819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[29819](info): set up 2 sockets
Aug 12 11:14:49 dist-test-slave-7z32 krb5kdc[29819](info): commencing operation
krb5kdc: starting...
W20260812 11:14:50.917472 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.373s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:14:51 dist-test-slave-7z32 krb5kdc[29819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:14:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:14:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:14:51.088215 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33537
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:51.226392 29839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:51.226788 29839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:51.226859 29839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:51.231447 29839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:14:51.231606 29839 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:14:51.231642 29839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:51.231685 29839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:51.231750 29839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:14:51.231791 29839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:14:51.240232 29839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33537
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:51.241717 29839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:51.242956 29839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:51.254647 29845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:51.254743 29849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:51.254691 29847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:51.254967 29839 server_base.cc:1034] running on GCE node
I20260812 11:14:51.256028 29839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:51.257468 29839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:51.258718 29839 hybrid_clock.cc:648] HybridClock initialized: now 1786533291258696 us; error 50 us; skew 500 ppm
Aug 12 11:14:51 dist-test-slave-7z32 krb5kdc[29819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:51.262497 29839 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:14:51.263978 29839 webserver.cc:460] Webserver started at http://127.27.109.62:39093/ using document root <none> and password file <none>
I20260812 11:14:51.264756 29839 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:51.264840 29839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:51.265069 29839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:51.268035 29839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "e63d449d4c964e40832f38da2f11db5d"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "b3e6d03e9fb0a6359c8870423484e35b"
server_key_iv: "1c855bae0beee5b8eebd3037830995de"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.268859 29839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "e63d449d4c964e40832f38da2f11db5d"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "b3e6d03e9fb0a6359c8870423484e35b"
server_key_iv: "1c855bae0beee5b8eebd3037830995de"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.275511 29839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 11:14:51.280445 29857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:51.283588 29839 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260812 11:14:51.283993 29839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "e63d449d4c964e40832f38da2f11db5d"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "b3e6d03e9fb0a6359c8870423484e35b"
server_key_iv: "1c855bae0beee5b8eebd3037830995de"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.284215 29839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:51.300465 29839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:51.305938 29839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:51.306274 29839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:51.320317 29839 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:33537
I20260812 11:14:51.320325 29958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:33537 every 8 connection(s)
I20260812 11:14:51.321834 29839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:14:51.325735 29960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:51.330181 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29839
I20260812 11:14:51.330340 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:14:51.330644 28084 external_mini_cluster.cc:1428] Setting key 99ccfa14b59a8c1fb6a25a681eaec971
I20260812 11:14:51.333264 29960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: Bootstrap starting.
I20260812 11:14:51.336395 29960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:51.337421 29960 log.cc:824] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:51.339844 29960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: No bootstrap required, opened a new log
Aug 12 11:14:51 dist-test-slave-7z32 krb5kdc[29819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:51.344287 29960 raft_consensus.cc:348] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63d449d4c964e40832f38da2f11db5d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33537 } }
I20260812 11:14:51.344725 29960 raft_consensus.cc:374] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:51.344838 29960 raft_consensus.cc:729] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e63d449d4c964e40832f38da2f11db5d, State: Initialized, Role: FOLLOWER
I20260812 11:14:51.345482 29960 consensus_queue.cc:260] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63d449d4c964e40832f38da2f11db5d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33537 } }
I20260812 11:14:51.345687 29960 raft_consensus.cc:388] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:14:51.345793 29960 raft_consensus.cc:482] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:14:51.345921 29960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:51.347229 29960 raft_consensus.cc:504] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63d449d4c964e40832f38da2f11db5d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33537 } }
I20260812 11:14:51.347635 29960 leader_election.cc:302] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e63d449d4c964e40832f38da2f11db5d; no voters:
I20260812 11:14:51.348099 29960 leader_election.cc:288] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:14:51.348254 29969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:51.348534 29969 raft_consensus.cc:686] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [term 1 LEADER]: Becoming Leader. State: Replica: e63d449d4c964e40832f38da2f11db5d, State: Running, Role: LEADER
I20260812 11:14:51.348965 29969 consensus_queue.cc:237] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63d449d4c964e40832f38da2f11db5d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33537 } }
I20260812 11:14:51.349727 29960 sys_catalog.cc:564] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:14:51.351060 29971 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e63d449d4c964e40832f38da2f11db5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63d449d4c964e40832f38da2f11db5d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33537 } } }
I20260812 11:14:51.351368 29971 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:51.351902 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:51.332888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.27.109.62:33537)
0812 11:14:51.333540 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:51.333562 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:51.333620 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:14:51.334517 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:51.334543 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:51.335112 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:51.335528 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:51.336757 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.338030 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.339192 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.339809 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.340024 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:51.343467 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:51.343475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:51.343501 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:51.343559 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:51.347012 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:51.347811 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:51.347818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:51.347825 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:51.348073 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:51.348433 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:51.348437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:51.348439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:51.349329 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:51.349623 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:51.349960 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 11:14:51.350425 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":220,"threads_started":1}
I20260812 11:14:51.352320 29970 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e63d449d4c964e40832f38da2f11db5d. Latest consensus state: current_term: 1 leader_uuid: "e63d449d4c964e40832f38da2f11db5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63d449d4c964e40832f38da2f11db5d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33537 } } }
I20260812 11:14:51.352926 29970 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d [sys.catalog]: This master's current role is: LEADER
W20260812 11:14:51.356705 29990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:14:51.356830 29990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:14:51.356930 29984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:14:51.357849 29984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:14:51.365134 29984 catalog_manager.cc:1347] Generated new cluster ID: 5c766d9cfbb54fafaf85aa7a21106a27
I20260812 11:14:51.365253 29984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:14:51.385628 29984 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:14:51.387077 29984 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:14:51.394400 29984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: Generated new TSK 0
I20260812 11:14:51.395533 29984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:14:51.439703 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33537
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:51.594183 30000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:51.594647 30000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:51.594743 30000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:51.600055 30000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:51.600450 30000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:51.600849 30000 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:14:51.600986 30000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:14:51.610574 30000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:33537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:51.612540 30000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:51.614737 30000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:51.633002 30006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:51.633002 30008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:51.633061 30005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:51.633349 30000 server_base.cc:1034] running on GCE node
I20260812 11:14:51.634220 30000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:51.635236 30000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:51.636478 30000 hybrid_clock.cc:648] HybridClock initialized: now 1786533291636446 us; error 57 us; skew 500 ppm
Aug 12 11:14:51 dist-test-slave-7z32 krb5kdc[29819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:51.640224 30000 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:14:51.641597 30000 webserver.cc:460] Webserver started at http://127.27.109.1:43399/ using document root <none> and password file <none>
I20260812 11:14:51.642345 30000 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:51.642454 30000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:51.642728 30000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:51.645946 30000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "30957af665ab480d824e86dafe4ef1e8"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "b9356323f7df358c3331c3195bacf23e"
server_key_iv: "dfa65f095525b9b17c66130a2890ada5"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.647313 30000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "30957af665ab480d824e86dafe4ef1e8"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "b9356323f7df358c3331c3195bacf23e"
server_key_iv: "dfa65f095525b9b17c66130a2890ada5"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.654428 30000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 11:14:51.658981 30022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.001s sys 0.000s
I20260812 11:14:51.661124 30000 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:14:51.661422 30000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "30957af665ab480d824e86dafe4ef1e8"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "b9356323f7df358c3331c3195bacf23e"
server_key_iv: "dfa65f095525b9b17c66130a2890ada5"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.661612 30000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:51.691131 30000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:51.694962 30000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:51.695155 30000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:51.696950 30000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:51.697021 30000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:51.697067 30000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:51.697084 30000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:51.710377 30000 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:41045
I20260812 11:14:51.710882 30199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:41045 every 8 connection(s)
I20260812 11:14:51.711805 30000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
Aug 12 11:14:51 dist-test-slave-7z32 krb5kdc[29819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:51.713366 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30000
I20260812 11:14:51.713511 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:14:51.713891 28084 external_mini_cluster.cc:1428] Setting key 931f4909ddf51fa6191be9337186d814
I20260812 11:14:51.719998 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:51.698868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50417 (local address 127.27.109.62:33537)
0812 11:14:51.699052 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:51.699057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:51.700456 (+ 1399us) server_negotiation.cc:407] Connection header received
0812 11:14:51.702339 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:51.702346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:51.702436 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:51.702590 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:51.704693 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.705577 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.707772 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.708162 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.708323 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:51.713238 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:51.713244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:51.713247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:51.713301 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:51.715325 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:51.716320 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:51.716326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:51.716328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:51.716481 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:51.717049 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:51.717055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:51.717060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:51.717439 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:51.717572 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:51.719470 (+ 1898us) server_negotiation.cc:299] Negotiation successful
0812 11:14:51.719724 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:14:51.720943 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:51.699335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33537 (local address 127.27.109.1:50417)
0812 11:14:51.700194 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:51.700278 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:51.701901 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:51.702903 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:51.702918 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:51.703589 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:51.704408 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:51.704433 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.705754 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.705758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:51.707449 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:51.707465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.708697 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.708705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:51.709057 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:51.710050 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:51.710083 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:51.712879 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:51.715546 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:51.715559 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:51.715583 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:51.716103 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:51.716642 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:51.716647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:51.716650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:51.716895 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:51.717650 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:51.717662 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:51.718160 (+ 498us) client_negotiation.cc:769] Sending connection context
0812 11:14:51.718615 (+ 455us) client_negotiation.cc:240] Negotiation successful
0812 11:14:51.719189 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":141,"threads_started":1}
I20260812 11:14:51.723105 30200 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33537
I20260812 11:14:51.723332 30200 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:51.724050 30200 heartbeater.cc:511] Master 127.27.109.62:33537 requested a full tablet report, sending...
I20260812 11:14:51.726445 29885 ts_manager.cc:194] Registered new tserver with Master: 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1:41045)
I20260812 11:14:51.728945 29885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:50417
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:14:51.739606 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:51.726724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33537 (local address 127.27.109.1:60535)
0812 11:14:51.729658 (+ 2934us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:51.729672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:51.729856 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:51.730190 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:51.730194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:51.730467 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:51.730636 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:51.730644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.732111 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.732115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:51.732771 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:51.732782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.733404 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.733411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:51.733553 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:51.734214 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:51.734227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:51.734751 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:51.737103 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:51.737111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:51.737116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:51.737522 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:51.737972 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:51.737978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:51.737983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:51.738107 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:51.738678 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:51.738683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:51.738932 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 11:14:51.739107 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 11:14:51.739360 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2840,"thread_start_us":210,"threads_started":1}
I20260812 11:14:51.739668 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:51.726941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:60535 (local address 127.27.109.62:33537)
0812 11:14:51.729322 (+ 2381us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:51.729328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:51.729778 (+ 450us) server_negotiation.cc:407] Connection header received
0812 11:14:51.729990 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:51.729995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:51.730056 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:51.730134 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:51.730813 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.731827 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.732931 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:51.733195 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:51.733355 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:51.735014 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:51.735018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:51.735021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:51.735052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:51.736946 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:51.737732 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:51.737736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:51.737737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:51.737843 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:51.738238 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:51.738242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:51.738243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:51.738503 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:51.738677 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:51.739182 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 11:14:51.739351 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2267}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:14:51.760545 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33537
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:51.901217 30212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:51.901523 30212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:51.901571 30212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:51.905663 30212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:51.905743 30212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:51.905791 30212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:14:51.905827 30212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:14:51.911602 30212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:33537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:51.912875 30212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:51.914041 30212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:51.927126 30218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:51.927151 30220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:51.927143 30222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:51.927557 30212 server_base.cc:1034] running on GCE node
I20260812 11:14:51.928241 30212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:51.929018 30212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:51.930265 30212 hybrid_clock.cc:648] HybridClock initialized: now 1786533291930209 us; error 73 us; skew 500 ppm
Aug 12 11:14:51 dist-test-slave-7z32 krb5kdc[29819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:51.934362 30212 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:14:51.935606 30212 webserver.cc:460] Webserver started at http://127.27.109.2:44209/ using document root <none> and password file <none>
I20260812 11:14:51.936266 30212 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:51.936365 30212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:51.936615 30212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:51.938977 30212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "bac7940483ee4ddabe577716373bda35"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "9aa5d8ebf03487d7aac5c8f1b8d9d3ab"
server_key_iv: "9bfbdd749a7ea0aab9f51941df8e3c2f"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.939602 30212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "bac7940483ee4ddabe577716373bda35"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "9aa5d8ebf03487d7aac5c8f1b8d9d3ab"
server_key_iv: "9bfbdd749a7ea0aab9f51941df8e3c2f"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.944523 30212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:14:51.948140 30235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:51.950157 30212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:14:51.950356 30212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "bac7940483ee4ddabe577716373bda35"
format_stamp: "Formatted at 2026-08-12 11:14:51 on dist-test-slave-7z32"
server_key: "9aa5d8ebf03487d7aac5c8f1b8d9d3ab"
server_key_iv: "9bfbdd749a7ea0aab9f51941df8e3c2f"
server_key_version: "encryptionkey@0"
I20260812 11:14:51.950512 30212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:52.007566 30212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:52.011287 30212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:52.011622 30212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:52.013422 30212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:52.013486 30212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:52.013535 30212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:52.013552 30212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:14:52 dist-test-slave-7z32 krb5kdc[29819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:52.026345 30212 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:39541
I20260812 11:14:52.026494 30410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:39541 every 8 connection(s)
I20260812 11:14:52.027685 30212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:14:52.032055 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.015317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:53065 (local address 127.27.109.62:33537)
0812 11:14:52.015490 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.015494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.016850 (+ 1356us) server_negotiation.cc:407] Connection header received
0812 11:14:52.018476 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.018481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.018570 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.018702 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:52.020319 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.020924 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.022853 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.023176 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.023308 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.026950 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:52.026955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:52.026958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:52.026990 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:52.029370 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.030116 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.030120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.030121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.030234 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.030630 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.030633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.030634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.030976 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:52.031097 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.031720 (+ 623us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.031884 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:14:52.032238 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30212
I20260812 11:14:52.032369 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:14:52.032701 28084 external_mini_cluster.cc:1428] Setting key b08ff2c1da1eadfd80efe2db92f3f981
I20260812 11:14:52.033156 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.015805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33537 (local address 127.27.109.2:53065)
0812 11:14:52.016579 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.016675 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.018118 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.018990 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.019000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.019483 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:52.020134 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.020151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.021088 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.021098 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.022682 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.022695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.023387 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.023391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.023615 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.024527 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:52.024555 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:52.026663 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:52.029545 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.029552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.029572 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.029966 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.030339 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.030341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.030342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.030512 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.031092 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:52.031098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:52.031390 (+ 292us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.031596 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.031941 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":181,"threads_started":1}
I20260812 11:14:52.034893 30411 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33537
I20260812 11:14:52.035140 30411 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:52.035909 30411 heartbeater.cc:511] Master 127.27.109.62:33537 requested a full tablet report, sending...
I20260812 11:14:52.037356 29885 ts_manager.cc:194] Registered new tserver with Master: bac7940483ee4ddabe577716373bda35 (127.27.109.2:39541)
I20260812 11:14:52.038368 29885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:53065
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:14:52.053869 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.037844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39093 (local address 127.27.109.62:33537)
0812 11:14:52.037996 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.037999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.043333 (+ 5334us) server_negotiation.cc:407] Connection header received
0812 11:14:52.043446 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.043453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.043549 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.043719 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:52.044649 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.045815 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.047076 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.047555 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.047724 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.049522 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:52.049526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:52.049528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:52.049565 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:52.051620 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.052326 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.052331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.052333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.052429 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.052748 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.052752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.052754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.053061 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:52.053204 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.053505 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.053639 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:14:52.053876 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.037750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33537 (local address 127.27.109.2:39093)
0812 11:14:52.043155 (+ 5405us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.043172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.043328 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.044050 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.044054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.044303 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:52.044440 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.044446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.046091 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.046095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.046798 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.046819 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.047793 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.047797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.047915 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.048678 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:52.048691 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:52.049190 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:52.051920 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.051925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.051927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.052202 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.052531 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.052535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.052537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.052631 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.053190 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:52.053193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:52.053312 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.053425 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.053651 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5284,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:14:52.076547 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33537
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:52.232466 30424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:52.232851 30424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:52.232908 30424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:52.237815 30424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:52.237931 30424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:52.237979 30424 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:14:52.238018 30424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:14:52.244999 30424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:33537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:52.246994 30424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:52.248432 30424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:52.261685 30435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:52.261700 30433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:52.261741 30424 server_base.cc:1034] running on GCE node
W20260812 11:14:52.262058 30437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:52.262593 30424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:52.263479 30424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:52.264711 30424 hybrid_clock.cc:648] HybridClock initialized: now 1786533292264689 us; error 48 us; skew 500 ppm
Aug 12 11:14:52 dist-test-slave-7z32 krb5kdc[29819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:52.268582 30424 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:14:52.269640 30424 webserver.cc:460] Webserver started at http://127.27.109.3:40413/ using document root <none> and password file <none>
I20260812 11:14:52.270339 30424 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:52.270449 30424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:52.270723 30424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:52.273118 30424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "f44696484e0d47b8967e142b4f9d3c0c"
format_stamp: "Formatted at 2026-08-12 11:14:52 on dist-test-slave-7z32"
server_key: "2c0e47a825c7a8878a626a4bf428725a"
server_key_iv: "054de496949c0f4fe11ee1a46fb9d5c7"
server_key_version: "encryptionkey@0"
I20260812 11:14:52.273757 30424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "f44696484e0d47b8967e142b4f9d3c0c"
format_stamp: "Formatted at 2026-08-12 11:14:52 on dist-test-slave-7z32"
server_key: "2c0e47a825c7a8878a626a4bf428725a"
server_key_iv: "054de496949c0f4fe11ee1a46fb9d5c7"
server_key_version: "encryptionkey@0"
I20260812 11:14:52.279276 30424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260812 11:14:52.283365 30447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:52.285373 30424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:14:52.285624 30424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "f44696484e0d47b8967e142b4f9d3c0c"
format_stamp: "Formatted at 2026-08-12 11:14:52 on dist-test-slave-7z32"
server_key: "2c0e47a825c7a8878a626a4bf428725a"
server_key_iv: "054de496949c0f4fe11ee1a46fb9d5c7"
server_key_version: "encryptionkey@0"
I20260812 11:14:52.285790 30424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:52.327610 30424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:52.331883 30424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:52.332161 30424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:52.334107 30424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:52.334213 30424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:52.334300 30424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:52.334352 30424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:14:52 dist-test-slave-7z32 krb5kdc[29819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:52.350095 30424 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:33887
I20260812 11:14:52.351953 30424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:14:52.350662 30621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:33887 every 8 connection(s)
I20260812 11:14:52.355314 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.335723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:47159 (local address 127.27.109.62:33537)
0812 11:14:52.335924 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.335931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.337218 (+ 1287us) server_negotiation.cc:407] Connection header received
0812 11:14:52.338793 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.338798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.338922 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.339029 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:52.340946 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.341559 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.343527 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.343961 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.344126 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.349019 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:52.349029 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:52.349036 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:52.349094 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:52.351797 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.353000 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.353008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.353010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.353164 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.353665 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.353668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.353670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.353973 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:52.354186 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.354890 (+ 704us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.355087 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:14:52.356285 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.336344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33537 (local address 127.27.109.3:47159)
0812 11:14:52.336996 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.337062 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.338486 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.339334 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.339344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.339899 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:52.340744 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.340764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.341721 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.341725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.343334 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.343355 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.344214 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.344219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.344719 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.345749 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:52.345780 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:52.348356 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:52.352123 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.352136 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.352167 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.352777 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.353346 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.353350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.353352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.353549 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.354170 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:52.354180 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:52.354520 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.354729 (+ 209us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.355260 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":202,"threads_started":1}
I20260812 11:14:52.358336 30622 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33537
I20260812 11:14:52.358590 30622 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:52.359360 30622 heartbeater.cc:511] Master 127.27.109.62:33537 requested a full tablet report, sending...
I20260812 11:14:52.359679 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30424
I20260812 11:14:52.359795 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:14:52.360113 28084 external_mini_cluster.cc:1428] Setting key 06246d820fed82ada0484061de025870
I20260812 11:14:52.360900 29883 ts_manager.cc:194] Registered new tserver with Master: f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3:33887)
I20260812 11:14:52.361814 29883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:47159
I20260812 11:14:52.362982 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:14:52.371642 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.361796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33537 (local address 127.27.109.3:34579)
0812 11:14:52.362237 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.362260 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.362443 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.363139 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.363143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.363421 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:52.363555 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.363564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.364641 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.364643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.365383 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.365400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.365932 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.365936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.366032 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.366713 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:52.366725 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:52.367254 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:52.369626 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.369630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.369632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.369936 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.370337 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.370340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.370342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.370441 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.370959 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:52.370962 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:52.371082 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.371206 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.371409 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":151,"threads_started":1}
I20260812 11:14:52.371637 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.361871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:34579 (local address 127.27.109.62:33537)
0812 11:14:52.362025 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.362031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.362348 (+ 317us) server_negotiation.cc:407] Connection header received
0812 11:14:52.362725 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.362729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.362799 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.362927 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:52.363707 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.364467 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.365509 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.365803 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.365924 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.367418 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:52.367422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:52.367424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:52.367454 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:52.369460 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.370077 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.370081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.370083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.370197 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.370550 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.370554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.370556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.370796 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:52.370961 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.371284 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.371422 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 11:14:52 dist-test-slave-7z32 krb5kdc[29819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:14:52 dist-test-slave-7z32 krb5kdc[29819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:52.404625 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.392115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.27.109.62:33537)
0812 11:14:52.392304 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.392308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.392463 (+ 155us) server_negotiation.cc:407] Connection header received
0812 11:14:52.392607 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.392611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.392684 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.392814 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:52.393880 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.394623 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.396045 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.396359 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.396497 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.400243 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:52.400248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:52.400251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:52.400303 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:52.402103 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.402741 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.402745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.402746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.402918 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.403292 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.403295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.403296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.403586 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:52.403710 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.404169 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.404379 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:14:52.410171 29885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:52.417146 29885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:52.437911 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.431381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33887 (local address 127.0.0.1:60584)
0812 11:14:52.431863 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.431878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.431981 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.432823 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.432826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.432874 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.433123 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.433134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.434740 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.434745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.436267 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.436278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.437238 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.437242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.437388 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.437444 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.437555 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.437706 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":101,"spinlock_wait_cycles":6784,"thread_start_us":159,"threads_started":1}
I20260812 11:14:52.438475 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.431479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39541 (local address 127.0.0.1:41068)
0812 11:14:52.432191 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.432210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.432333 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.433750 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.433754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.433773 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.433872 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.433878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.435311 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.435316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.436747 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.436760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.438039 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.438045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.438151 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.438174 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.438264 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.438348 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":143,"threads_started":1}
I20260812 11:14:52.438701 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.431655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.27.109.3:33887)
0812 11:14:52.432105 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.432121 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.432164 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:14:52.432286 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.432293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.432591 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.432827 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.433309 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.434591 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.436404 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.437083 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.438159 (+ 1076us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.438364 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.438422 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.438539 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":194,"threads_started":1}
I20260812 11:14:52.439026 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.431830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.27.109.2:39541)
0812 11:14:52.433131 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.433143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.433168 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:14:52.433246 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.433250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.433434 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.433590 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.434000 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.435173 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.436891 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.437676 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.437776 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.437962 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.438746 (+ 784us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.438905 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":140,"threads_started":1}
I20260812 11:14:52.439148 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.430270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41045 (local address 127.0.0.1:44838)
0812 11:14:52.431104 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.431211 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.431382 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.432815 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.432823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.432872 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.433145 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.433155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.434694 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.434699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.436631 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.436646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.438520 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.438524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.438652 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.438695 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.438795 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.438948 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":237,"spinlock_wait_cycles":7424,"thread_start_us":186,"threads_started":1}
I20260812 11:14:52.439716 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.430491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.27.109.1:41045)
0812 11:14:52.432248 (+ 1757us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.432255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.432279 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:14:52.432422 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.432427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.432622 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.432785 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.433341 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.434550 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.437086 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.438087 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.438222 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.438444 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.439402 (+ 958us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.439543 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1629,"thread_start_us":136,"threads_started":1}
I20260812 11:14:52.442477 30301 tablet_service.cc:1467] Processing CreateTablet for tablet 519c709ac1774caab6f3ed04e562417a (DEFAULT_TABLE table=test-table [id=4fd03d5f9c7e4bf685acf0d4762a8b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:52.442477 30094 tablet_service.cc:1467] Processing CreateTablet for tablet 519c709ac1774caab6f3ed04e562417a (DEFAULT_TABLE table=test-table [id=4fd03d5f9c7e4bf685acf0d4762a8b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:52.442901 30520 tablet_service.cc:1467] Processing CreateTablet for tablet 519c709ac1774caab6f3ed04e562417a (DEFAULT_TABLE table=test-table [id=4fd03d5f9c7e4bf685acf0d4762a8b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:52.443868 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519c709ac1774caab6f3ed04e562417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.443868 30301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519c709ac1774caab6f3ed04e562417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.444501 30520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519c709ac1774caab6f3ed04e562417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.454609 30665 tablet_bootstrap.cc:492] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Bootstrap starting.
I20260812 11:14:52.455621 30666 tablet_bootstrap.cc:492] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Bootstrap starting.
I20260812 11:14:52.458021 30665 tablet_bootstrap.cc:654] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.458483 30666 tablet_bootstrap.cc:654] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.459299 30665 log.cc:824] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:52.459509 30666 log.cc:824] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:52.460026 30664 tablet_bootstrap.cc:492] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Bootstrap starting.
I20260812 11:14:52.462317 30666 tablet_bootstrap.cc:492] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: No bootstrap required, opened a new log
I20260812 11:14:52.462625 30666 ts_tablet_manager.cc:1397] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:14:52.463995 30664 tablet_bootstrap.cc:654] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.464861 30664 log.cc:824] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:52.467259 30665 tablet_bootstrap.cc:492] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: No bootstrap required, opened a new log
I20260812 11:14:52.467535 30665 ts_tablet_manager.cc:1397] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260812 11:14:52.467471 30666 raft_consensus.cc:348] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.467887 30666 raft_consensus.cc:374] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.467970 30666 raft_consensus.cc:729] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44696484e0d47b8967e142b4f9d3c0c, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.468052 30664 tablet_bootstrap.cc:492] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: No bootstrap required, opened a new log
I20260812 11:14:52.468300 30664 ts_tablet_manager.cc:1397] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260812 11:14:52.468677 30666 consensus_queue.cc:260] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.469801 30622 heartbeater.cc:503] Master 127.27.109.62:33537 was elected leader, sending a full tablet report...
I20260812 11:14:52.470165 30666 ts_tablet_manager.cc:1428] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260812 11:14:52.473207 30665 raft_consensus.cc:348] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.473510 30665 raft_consensus.cc:374] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.473548 30665 raft_consensus.cc:729] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30957af665ab480d824e86dafe4ef1e8, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.473500 30664 raft_consensus.cc:348] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.473869 30664 raft_consensus.cc:374] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.473928 30664 raft_consensus.cc:729] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac7940483ee4ddabe577716373bda35, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.474057 30665 consensus_queue.cc:260] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.474469 30664 consensus_queue.cc:260] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.475417 30665 ts_tablet_manager.cc:1428] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 11:14:52.475481 30411 heartbeater.cc:503] Master 127.27.109.62:33537 was elected leader, sending a full tablet report...
I20260812 11:14:52.475759 30200 heartbeater.cc:503] Master 127.27.109.62:33537 was elected leader, sending a full tablet report...
I20260812 11:14:52.475831 30664 ts_tablet_manager.cc:1428] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260812 11:14:52.529690 30413 tablet.cc:2369] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:14:52.553560 30673 raft_consensus.cc:482] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:52.553809 30673 raft_consensus.cc:504] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.555495 30673 leader_election.cc:288] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac7940483ee4ddabe577716373bda35 (127.27.109.2:39541), f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3:33887)
I20260812 11:14:52.561537 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.556371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33711 (local address 127.27.109.3:33887)
0812 11:14:52.556520 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.556525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.556762 (+ 237us) server_negotiation.cc:407] Connection header received
0812 11:14:52.556956 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.556959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.557009 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.557094 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.557502 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.558199 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.560064 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.560767 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.560932 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.561035 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.561244 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.561349 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:14:52.561625 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.556118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33887 (local address 127.27.109.1:33711)
0812 11:14:52.556583 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.556603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.556798 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.557133 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.557135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.557153 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.557325 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.557332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.558365 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.558369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.559853 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.559860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.560977 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.560981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.561080 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.561111 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.561199 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.561284 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":96,"thread_start_us":205,"threads_started":1}
I20260812 11:14:52.562772 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.555748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39541 (local address 127.27.109.1:57169)
0812 11:14:52.556346 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.556363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.556558 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.556957 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.556960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.556983 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.557166 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.557177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.558266 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.558273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.559847 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.559860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.562181 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.562186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.562347 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.562380 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.562485 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.562578 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":86,"thread_start_us":175,"threads_started":1}
I20260812 11:14:52.562769 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.555921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57169 (local address 127.27.109.2:39541)
0812 11:14:52.556163 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.556168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.556457 (+ 289us) server_negotiation.cc:407] Connection header received
0812 11:14:52.556757 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.556763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.556825 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.556926 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.557314 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.558136 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.561060 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.561760 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.561923 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.562019 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.562545 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.562662 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 11:14:52.562695 30525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c709ac1774caab6f3ed04e562417a" candidate_uuid: "30957af665ab480d824e86dafe4ef1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44696484e0d47b8967e142b4f9d3c0c" is_pre_election: true
I20260812 11:14:52.563169 30525 raft_consensus.cc:2454] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30957af665ab480d824e86dafe4ef1e8 in term 0.
I20260812 11:14:52.563625 30336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c709ac1774caab6f3ed04e562417a" candidate_uuid: "30957af665ab480d824e86dafe4ef1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac7940483ee4ddabe577716373bda35" is_pre_election: true
I20260812 11:14:52.563863 30028 leader_election.cc:302] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30957af665ab480d824e86dafe4ef1e8, f44696484e0d47b8967e142b4f9d3c0c; no voters:
I20260812 11:14:52.563956 30336 raft_consensus.cc:2454] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30957af665ab480d824e86dafe4ef1e8 in term 0.
I20260812 11:14:52.564271 30673 raft_consensus.cc:2781] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:52.564443 30673 raft_consensus.cc:482] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:52.564515 30673 raft_consensus.cc:3037] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.565932 30673 raft_consensus.cc:504] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.566414 30673 leader_election.cc:288] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [CANDIDATE]: Term 1 election: Requested vote from peers bac7940483ee4ddabe577716373bda35 (127.27.109.2:39541), f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3:33887)
I20260812 11:14:52.567294 30336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c709ac1774caab6f3ed04e562417a" candidate_uuid: "30957af665ab480d824e86dafe4ef1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac7940483ee4ddabe577716373bda35"
I20260812 11:14:52.567294 30525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c709ac1774caab6f3ed04e562417a" candidate_uuid: "30957af665ab480d824e86dafe4ef1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44696484e0d47b8967e142b4f9d3c0c"
I20260812 11:14:52.567517 30336 raft_consensus.cc:3037] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.567517 30525 raft_consensus.cc:3037] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.569558 30336 raft_consensus.cc:2454] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30957af665ab480d824e86dafe4ef1e8 in term 1.
I20260812 11:14:52.569809 30525 raft_consensus.cc:2454] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30957af665ab480d824e86dafe4ef1e8 in term 1.
I20260812 11:14:52.570156 30026 leader_election.cc:302] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30957af665ab480d824e86dafe4ef1e8, bac7940483ee4ddabe577716373bda35; no voters:
I20260812 11:14:52.570443 30673 raft_consensus.cc:2781] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:52.570868 30673 raft_consensus.cc:686] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 30957af665ab480d824e86dafe4ef1e8, State: Running, Role: LEADER
I20260812 11:14:52.571324 30673 consensus_queue.cc:237] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.575403 29885 catalog_manager.cc:5495] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "30957af665ab480d824e86dafe4ef1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } health_report { overall_health: UNKNOWN } } }
W20260812 11:14:52.603533 30623 tablet.cc:2369] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:14:52.609277 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.604122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.27.109.1:41045)
0812 11:14:52.604301 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.604307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.604333 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:14:52.604641 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.604644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.604714 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.604882 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:52.605384 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.605952 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.607229 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.607521 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.607644 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.608178 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:52.608757 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:52.608869 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.609071 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.609148 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:14:52.617894 30525 raft_consensus.cc:1261] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Refusing update from remote peer 30957af665ab480d824e86dafe4ef1e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:52.617894 30336 raft_consensus.cc:1261] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Refusing update from remote peer 30957af665ab480d824e86dafe4ef1e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:52.619181 30673 consensus_queue.cc:1035] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:52.619683 30681 consensus_queue.cc:1035] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:52.649595 29889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:52.649935 29889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:52.654253 29889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: Sending DeleteTablet for 3 replicas of tablet 519c709ac1774caab6f3ed04e562417a
I20260812 11:14:52.655753 30301 tablet_service.cc:1514] Processing DeleteTablet for tablet 519c709ac1774caab6f3ed04e562417a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:52 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:41068
I20260812 11:14:52.655761 30094 tablet_service.cc:1514] Processing DeleteTablet for tablet 519c709ac1774caab6f3ed04e562417a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:52 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:44838
I20260812 11:14:52.656570 30520 tablet_service.cc:1514] Processing DeleteTablet for tablet 519c709ac1774caab6f3ed04e562417a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:52 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:60584
I20260812 11:14:52.656621 30705 tablet_replica.cc:330] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: stopping tablet replica
I20260812 11:14:52.657087 30705 raft_consensus.cc:2229] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:14:52.657783 30705 raft_consensus.cc:2258] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:52.657835 30706 tablet_replica.cc:330] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: stopping tablet replica
I20260812 11:14:52.658550 30706 raft_consensus.cc:2229] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:52.659096 30706 raft_consensus.cc:2258] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:52.659816 30705 ts_tablet_manager.cc:1905] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:52.659873 29889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:14:52.660123 30704 tablet_replica.cc:330] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: stopping tablet replica
I20260812 11:14:52.660503 30704 raft_consensus.cc:2229] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 11:14:52.660547 29889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:52.660907 30704 raft_consensus.cc:2258] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:52.663154 30704 ts_tablet_manager.cc:1905] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:52.663446 30705 ts_tablet_manager.cc:1918] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:52.663594 30705 log.cc:1196] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/519c709ac1774caab6f3ed04e562417a
I20260812 11:14:52.663949 30705 ts_tablet_manager.cc:1939] T 519c709ac1774caab6f3ed04e562417a P 30957af665ab480d824e86dafe4ef1e8: Deleting consensus metadata
I20260812 11:14:52.665969 29864 catalog_manager.cc:4841] TS 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1:41045): tablet 519c709ac1774caab6f3ed04e562417a (table test-table [id=4fd03d5f9c7e4bf685acf0d4762a8b45]) successfully deleted
I20260812 11:14:52.667160 30706 ts_tablet_manager.cc:1905] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:52.669497 30301 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac839017bab4fddb099023a1efa8866 (DEFAULT_TABLE table=test-table [id=22f18ef0a37043a6b4a6c25b9b2acba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:52.669497 30520 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac839017bab4fddb099023a1efa8866 (DEFAULT_TABLE table=test-table [id=22f18ef0a37043a6b4a6c25b9b2acba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:52.669881 30520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac839017bab4fddb099023a1efa8866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.669881 30301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac839017bab4fddb099023a1efa8866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.670296 30706 ts_tablet_manager.cc:1918] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:52.670411 30706 log.cc:1196] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/519c709ac1774caab6f3ed04e562417a
I20260812 11:14:52.670773 30706 ts_tablet_manager.cc:1939] T 519c709ac1774caab6f3ed04e562417a P f44696484e0d47b8967e142b4f9d3c0c: Deleting consensus metadata
I20260812 11:14:52.671459 30094 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac839017bab4fddb099023a1efa8866 (DEFAULT_TABLE table=test-table [id=22f18ef0a37043a6b4a6c25b9b2acba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:52.671802 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac839017bab4fddb099023a1efa8866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.672374 29866 catalog_manager.cc:4841] TS f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3:33887): tablet 519c709ac1774caab6f3ed04e562417a (table test-table [id=4fd03d5f9c7e4bf685acf0d4762a8b45]) successfully deleted
I20260812 11:14:52.674007 30666 tablet_bootstrap.cc:492] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: Bootstrap starting.
I20260812 11:14:52.674527 30704 ts_tablet_manager.cc:1918] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:52.674685 30704 log.cc:1196] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/519c709ac1774caab6f3ed04e562417a
I20260812 11:14:52.675117 30704 ts_tablet_manager.cc:1939] T 519c709ac1774caab6f3ed04e562417a P bac7940483ee4ddabe577716373bda35: Deleting consensus metadata
I20260812 11:14:52.675113 30665 tablet_bootstrap.cc:492] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Bootstrap starting.
I20260812 11:14:52.675383 30666 tablet_bootstrap.cc:654] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.676462 30665 tablet_bootstrap.cc:654] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.676822 29864 catalog_manager.cc:4841] TS bac7940483ee4ddabe577716373bda35 (127.27.109.2:39541): tablet 519c709ac1774caab6f3ed04e562417a (table test-table [id=4fd03d5f9c7e4bf685acf0d4762a8b45]) successfully deleted
I20260812 11:14:52.678340 30666 tablet_bootstrap.cc:492] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: No bootstrap required, opened a new log
I20260812 11:14:52.678359 30665 tablet_bootstrap.cc:492] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: No bootstrap required, opened a new log
I20260812 11:14:52.678458 30664 tablet_bootstrap.cc:492] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: Bootstrap starting.
I20260812 11:14:52.678469 30665 ts_tablet_manager.cc:1397] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:52.678522 30666 ts_tablet_manager.cc:1397] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:14:52.679141 30665 raft_consensus.cc:348] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.679196 30666 raft_consensus.cc:348] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.679314 30665 raft_consensus.cc:374] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.679351 30665 raft_consensus.cc:729] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30957af665ab480d824e86dafe4ef1e8, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.679333 30666 raft_consensus.cc:374] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.679498 30666 raft_consensus.cc:729] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44696484e0d47b8967e142b4f9d3c0c, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.679509 30665 consensus_queue.cc:260] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.679786 30664 tablet_bootstrap.cc:654] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.679841 30665 ts_tablet_manager.cc:1428] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:14:52.679780 30666 consensus_queue.cc:260] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.682256 30664 tablet_bootstrap.cc:492] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: No bootstrap required, opened a new log
I20260812 11:14:52.682374 30664 ts_tablet_manager.cc:1397] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 11:14:52.682974 30664 raft_consensus.cc:348] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.683099 30666 ts_tablet_manager.cc:1428] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260812 11:14:52.683128 30664 raft_consensus.cc:374] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.683164 30664 raft_consensus.cc:729] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac7940483ee4ddabe577716373bda35, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.683329 30664 consensus_queue.cc:260] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.684119 30664 ts_tablet_manager.cc:1428] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260812 11:14:52.716864 30201 tablet.cc:2369] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:14:52.783624 30672 raft_consensus.cc:482] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:52.783823 30672 raft_consensus.cc:504] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.785418 30672 leader_election.cc:288] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1:41045), bac7940483ee4ddabe577716373bda35 (127.27.109.2:39541)
I20260812 11:14:52.791570 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.786469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:32847 (local address 127.27.109.2:39541)
0812 11:14:52.786618 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.786623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.786727 (+ 104us) server_negotiation.cc:407] Connection header received
0812 11:14:52.786871 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.786875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.786943 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.787091 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.787433 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.788220 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.790018 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.790692 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.790949 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.791024 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.791264 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.791398 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:14:52.791589 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.785744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41045 (local address 127.27.109.3:40599)
0812 11:14:52.786187 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.786212 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.786362 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.786924 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.786927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.786943 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.787115 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.787123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.788090 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.788097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.789675 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.789687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.791181 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.791185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.791294 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.791314 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.791389 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.791452 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260812 11:14:52.791605 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.786103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39541 (local address 127.27.109.3:32847)
0812 11:14:52.786587 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.786602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.786730 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.787117 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.787120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.787136 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.787294 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.787305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.788376 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.788379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.789853 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.789863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.790961 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.790965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.791085 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.791112 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.791226 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.791405 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":158,"threads_started":1}
I20260812 11:14:52.792073 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.785970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:40599 (local address 127.27.109.1:41045)
0812 11:14:52.786193 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.786199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.786476 (+ 277us) server_negotiation.cc:407] Connection header received
0812 11:14:52.786556 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.786560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.786637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.786722 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.787274 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.787947 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.789868 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.790789 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.790995 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.791064 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.791831 (+ 767us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.791922 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 11:14:52.792752 30123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac839017bab4fddb099023a1efa8866" candidate_uuid: "f44696484e0d47b8967e142b4f9d3c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30957af665ab480d824e86dafe4ef1e8" is_pre_election: true
I20260812 11:14:52.793084 30123 raft_consensus.cc:2454] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f44696484e0d47b8967e142b4f9d3c0c in term 0.
I20260812 11:14:52.793046 30336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac839017bab4fddb099023a1efa8866" candidate_uuid: "f44696484e0d47b8967e142b4f9d3c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac7940483ee4ddabe577716373bda35" is_pre_election: true
I20260812 11:14:52.793282 30336 raft_consensus.cc:2454] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f44696484e0d47b8967e142b4f9d3c0c in term 0.
I20260812 11:14:52.793687 30455 leader_election.cc:302] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30957af665ab480d824e86dafe4ef1e8, f44696484e0d47b8967e142b4f9d3c0c; no voters:
I20260812 11:14:52.794085 30672 raft_consensus.cc:2781] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:52.794193 30672 raft_consensus.cc:482] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:52.794315 30672 raft_consensus.cc:3037] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.795898 30672 raft_consensus.cc:504] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.796378 30672 leader_election.cc:288] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [CANDIDATE]: Term 1 election: Requested vote from peers 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1:41045), bac7940483ee4ddabe577716373bda35 (127.27.109.2:39541)
I20260812 11:14:52.797039 30336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac839017bab4fddb099023a1efa8866" candidate_uuid: "f44696484e0d47b8967e142b4f9d3c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac7940483ee4ddabe577716373bda35"
I20260812 11:14:52.797052 30123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac839017bab4fddb099023a1efa8866" candidate_uuid: "f44696484e0d47b8967e142b4f9d3c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30957af665ab480d824e86dafe4ef1e8"
I20260812 11:14:52.797212 30123 raft_consensus.cc:3037] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.797224 30336 raft_consensus.cc:3037] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.798477 30336 raft_consensus.cc:2454] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f44696484e0d47b8967e142b4f9d3c0c in term 1.
I20260812 11:14:52.798952 30455 leader_election.cc:302] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bac7940483ee4ddabe577716373bda35, f44696484e0d47b8967e142b4f9d3c0c; no voters:
I20260812 11:14:52.799206 30672 raft_consensus.cc:2781] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:52.799258 30123 raft_consensus.cc:2454] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f44696484e0d47b8967e142b4f9d3c0c in term 1.
I20260812 11:14:52.799535 30672 raft_consensus.cc:686] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 1 LEADER]: Becoming Leader. State: Replica: f44696484e0d47b8967e142b4f9d3c0c, State: Running, Role: LEADER
I20260812 11:14:52.799939 30672 consensus_queue.cc:237] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.803398 29885 catalog_manager.cc:5495] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c reported cstate change: term changed from 0 to 1, leader changed from <none> to f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3). New cstate: current_term: 1 leader_uuid: "f44696484e0d47b8967e142b4f9d3c0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } health_report { overall_health: HEALTHY } } }
Aug 12 11:14:52 dist-test-slave-7z32 krb5kdc[29819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:52.858736 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.846102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33537 (local address 127.0.0.1:57132)
0812 11:14:52.846719 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.846736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.847137 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.847537 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.847540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.848109 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:52.848208 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.848214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.849039 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.849046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.849618 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.849626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.850115 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.850118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.850190 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.851230 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:52.851243 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:52.853506 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:52.856180 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.856201 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.856213 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.856750 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.857221 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:52.857224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:52.857226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:52.857353 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:52.857915 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:52.857923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:52.858162 (+ 239us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.858343 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.858545 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":257,"threads_started":1}
I20260812 11:14:52.858886 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.846286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.27.109.62:33537)
0812 11:14:52.846614 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.846621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.846880 (+ 259us) server_negotiation.cc:407] Connection header received
0812 11:14:52.847321 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.847325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.847379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.847482 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:52.848338 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.848898 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.849754 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.849979 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.850094 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.853730 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:52.853735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:52.853738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:52.853775 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:52.855941 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.856950 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.856954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.856955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.857061 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:52.857496 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:52.857499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:52.857500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:52.857740 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:52.857900 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.858475 (+ 575us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.858628 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260812 11:14:52.864674 29889 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57132:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:14:52.865797 29889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:52.873942 30301 tablet_service.cc:1467] Processing CreateTablet for tablet 146f85b9343748729a5181f32d1eaaee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf5e6ae14e5c4b819e09a969a86f2b0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:14:52.873942 30094 tablet_service.cc:1467] Processing CreateTablet for tablet 146f85b9343748729a5181f32d1eaaee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf5e6ae14e5c4b819e09a969a86f2b0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:14:52.874135 30520 tablet_service.cc:1467] Processing CreateTablet for tablet 146f85b9343748729a5181f32d1eaaee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf5e6ae14e5c4b819e09a969a86f2b0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:14:52.874594 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146f85b9343748729a5181f32d1eaaee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.874594 30301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146f85b9343748729a5181f32d1eaaee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.874893 30520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146f85b9343748729a5181f32d1eaaee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:52.878280 30666 tablet_bootstrap.cc:492] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: Bootstrap starting.
I20260812 11:14:52.878758 30665 tablet_bootstrap.cc:492] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: Bootstrap starting.
I20260812 11:14:52.879087 30664 tablet_bootstrap.cc:492] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: Bootstrap starting.
I20260812 11:14:52.879586 30666 tablet_bootstrap.cc:654] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.880028 30665 tablet_bootstrap.cc:654] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.880333 30664 tablet_bootstrap.cc:654] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:52.881412 30666 tablet_bootstrap.cc:492] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: No bootstrap required, opened a new log
I20260812 11:14:52.881505 30666 ts_tablet_manager.cc:1397] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:52.881875 30665 tablet_bootstrap.cc:492] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: No bootstrap required, opened a new log
I20260812 11:14:52.882011 30665 ts_tablet_manager.cc:1397] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 11:14:52.882040 30666 raft_consensus.cc:348] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.882208 30666 raft_consensus.cc:374] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.882236 30666 raft_consensus.cc:729] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44696484e0d47b8967e142b4f9d3c0c, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.882424 30666 consensus_queue.cc:260] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.882642 30665 raft_consensus.cc:348] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.882777 30666 ts_tablet_manager.cc:1428] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:14:52.882815 30665 raft_consensus.cc:374] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.882930 30665 raft_consensus.cc:729] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30957af665ab480d824e86dafe4ef1e8, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.883104 30665 consensus_queue.cc:260] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.883415 30665 ts_tablet_manager.cc:1428] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 11:14:52.883824 30672 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.883999 30672 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:52.884248 30697 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.884411 30697 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:52.885208 30664 tablet_bootstrap.cc:492] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: No bootstrap required, opened a new log
I20260812 11:14:52.885349 30664 ts_tablet_manager.cc:1397] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 11:14:52.885953 30664 raft_consensus.cc:348] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.886075 30664 raft_consensus.cc:374] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:52.886126 30664 raft_consensus.cc:729] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac7940483ee4ddabe577716373bda35, State: Initialized, Role: FOLLOWER
I20260812 11:14:52.886338 30664 consensus_queue.cc:260] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.886664 30674 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.886875 30674 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:52.887730 30664 ts_tablet_manager.cc:1428] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:52.897419 30674 raft_consensus.cc:482] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:52.897635 30674 raft_consensus.cc:504] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.899448 30674 leader_election.cc:288] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1:41045), f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3:33887)
I20260812 11:14:52.906536 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.899991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:49771 (local address 127.27.109.3:33887)
0812 11:14:52.900237 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.900246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.900608 (+ 362us) server_negotiation.cc:407] Connection header received
0812 11:14:52.900805 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.900809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.900876 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.900983 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.901922 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.902703 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.904978 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.905654 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.905812 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.905886 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.906281 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.906365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 11:14:52.906579 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.899836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33887 (local address 127.27.109.2:49771)
0812 11:14:52.900456 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.900475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.900658 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.901060 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.901064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.901123 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.901586 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.901598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.902875 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.902878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.904764 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.904783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.905987 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.905991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.906119 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.906150 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.906259 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.906373 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":335,"spinlock_wait_cycles":45696,"thread_start_us":225,"threads_started":1}
I20260812 11:14:52.906929 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.900210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:42107 (local address 127.27.109.1:41045)
0812 11:14:52.900373 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.900378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.900691 (+ 313us) server_negotiation.cc:407] Connection header received
0812 11:14:52.900833 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.900836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.900881 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.900983 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:52.901757 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.902622 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.905026 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.905664 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.905825 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.905900 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.906412 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.906525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:14:52.906997 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.900031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41045 (local address 127.27.109.2:42107)
0812 11:14:52.900564 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.900579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.900684 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.901060 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.901064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.901124 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:52.901517 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.901528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.902830 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.902834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.904764 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.904783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.905820 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.905824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.906253 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.906286 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.906389 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.906496 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":253,"spinlock_wait_cycles":8192,"thread_start_us":158,"threads_started":1}
I20260812 11:14:52.907261 30525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146f85b9343748729a5181f32d1eaaee" candidate_uuid: "bac7940483ee4ddabe577716373bda35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44696484e0d47b8967e142b4f9d3c0c" is_pre_election: true
I20260812 11:14:52.907497 30525 raft_consensus.cc:2454] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac7940483ee4ddabe577716373bda35 in term 0.
I20260812 11:14:52.907648 30123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146f85b9343748729a5181f32d1eaaee" candidate_uuid: "bac7940483ee4ddabe577716373bda35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30957af665ab480d824e86dafe4ef1e8" is_pre_election: true
I20260812 11:14:52.907886 30123 raft_consensus.cc:2454] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac7940483ee4ddabe577716373bda35 in term 0.
I20260812 11:14:52.908033 30240 leader_election.cc:302] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bac7940483ee4ddabe577716373bda35, f44696484e0d47b8967e142b4f9d3c0c; no voters:
I20260812 11:14:52.908618 30674 raft_consensus.cc:2781] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:52.908716 30674 raft_consensus.cc:482] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:52.908764 30674 raft_consensus.cc:3037] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.910254 30674 raft_consensus.cc:504] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.910977 30674 leader_election.cc:288] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [CANDIDATE]: Term 1 election: Requested vote from peers 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1:41045), f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3:33887)
I20260812 11:14:52.911491 30123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146f85b9343748729a5181f32d1eaaee" candidate_uuid: "bac7940483ee4ddabe577716373bda35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30957af665ab480d824e86dafe4ef1e8"
I20260812 11:14:52.911687 30123 raft_consensus.cc:3037] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.911644 30525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146f85b9343748729a5181f32d1eaaee" candidate_uuid: "bac7940483ee4ddabe577716373bda35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44696484e0d47b8967e142b4f9d3c0c"
I20260812 11:14:52.911859 30525 raft_consensus.cc:3037] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:52.912969 30123 raft_consensus.cc:2454] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac7940483ee4ddabe577716373bda35 in term 1.
I20260812 11:14:52.913308 30525 raft_consensus.cc:2454] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac7940483ee4ddabe577716373bda35 in term 1.
I20260812 11:14:52.913424 30239 leader_election.cc:302] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30957af665ab480d824e86dafe4ef1e8, bac7940483ee4ddabe577716373bda35; no voters:
I20260812 11:14:52.913691 30674 raft_consensus.cc:2781] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:52.913970 30674 raft_consensus.cc:686] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [term 1 LEADER]: Becoming Leader. State: Replica: bac7940483ee4ddabe577716373bda35, State: Running, Role: LEADER
I20260812 11:14:52.914331 30674 consensus_queue.cc:237] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } }
I20260812 11:14:52.916054 30739 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: TxnStatusTablet state changed. Reason: New leader bac7940483ee4ddabe577716373bda35. Latest consensus state: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.916292 30739 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:14:52.916815 30742 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:14:52.917088 30742 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:14:52.917959 29885 catalog_manager.cc:5495] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 reported cstate change: term changed from 0 to 1, leader changed from <none> to bac7940483ee4ddabe577716373bda35 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } health_report { overall_health: UNKNOWN } } }
I20260812 11:14:52.940131 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.934275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39541 (local address 127.0.0.1:41070)
0812 11:14:52.934615 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.934637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.934864 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.935635 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.935638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.935646 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:14:52.935801 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.935808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.936970 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.936973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.937645 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.937655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.938366 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.938370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.938496 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.938557 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:14:52.939775 (+ 1218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:14:52.939803 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.939881 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.939992 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260812 11:14:52.940274 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.934926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.27.109.2:39541)
0812 11:14:52.935140 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.935147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.935172 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:14:52.935347 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.935351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.935440 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.935583 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:52.935942 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.936810 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.937786 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.938156 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.938301 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.938755 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:52.939639 (+ 884us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:52.939779 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.939930 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.940019 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:14:52.944422 30525 raft_consensus.cc:1261] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Refusing update from remote peer bac7940483ee4ddabe577716373bda35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:52.944489 30123 raft_consensus.cc:1261] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Refusing update from remote peer bac7940483ee4ddabe577716373bda35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:52.945178 30739 consensus_queue.cc:1035] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:52.945433 30674 consensus_queue.cc:1035] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:52.948088 30697 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: TxnStatusTablet state changed. Reason: New leader bac7940483ee4ddabe577716373bda35. Latest consensus state: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.948241 30697 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:52.948523 30672 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: TxnStatusTablet state changed. Reason: New leader bac7940483ee4ddabe577716373bda35. Latest consensus state: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.948722 30672 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:52.951830 30739 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.952000 30739 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:14:52.952131 30687 mvcc.cc:205] Tried to move back new op lower bound from 7317640367893118976 to 7317640367780786176. Current Snapshot: MvccSnapshot[applied={T|T < 7317640367893118976}]
I20260812 11:14:52.954118 30674 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.954346 30674 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P bac7940483ee4ddabe577716373bda35: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:14:52.957103 30672 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.957325 30672 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P f44696484e0d47b8967e142b4f9d3c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:52.959923 30697 tablet_replica.cc:439] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bac7940483ee4ddabe577716373bda35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 } } peers { permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 } } peers { permanent_uuid: "f44696484e0d47b8967e142b4f9d3c0c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33887 } } }
I20260812 11:14:52.960103 30697 tablet_replica.cc:442] T 146f85b9343748729a5181f32d1eaaee P 30957af665ab480d824e86dafe4ef1e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:52.965042 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.959165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.27.109.3:33887)
0812 11:14:52.959307 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.959312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.959333 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:14:52.959527 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.959530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.959623 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.959772 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:52.960497 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.961435 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.963021 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.963464 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.963574 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.963968 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:52.964502 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:52.964598 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.964852 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.964923 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:14:52.982481 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.976982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39541 (local address 127.27.109.3:57905)
0812 11:14:52.977404 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:52.977421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:52.977545 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:52.977941 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:52.977943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:52.977952 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:14:52.978094 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.978099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.979459 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.979462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.980864 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:52.980875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.981567 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.981569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:52.981644 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.981712 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:14:52.982145 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:14:52.982154 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:14:52.982249 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 11:14:52.982325 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":164,"threads_started":1}
I20260812 11:14:52.982625 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:52.977208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57905 (local address 127.27.109.2:39541)
0812 11:14:52.977419 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:52.977425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:52.977537 (+ 112us) server_negotiation.cc:407] Connection header received
0812 11:14:52.977665 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:52.977670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:52.977744 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:52.977882 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:52.978248 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.979273 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.981019 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:52.981427 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:52.981567 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:52.981888 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:52.982018 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:52.982126 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:52.982294 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 11:14:52.982383 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 11:14:52.991631 30123 raft_consensus.cc:1261] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Refusing update from remote peer f44696484e0d47b8967e142b4f9d3c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:52.991719 30336 raft_consensus.cc:1261] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Refusing update from remote peer f44696484e0d47b8967e142b4f9d3c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:52.992230 30672 consensus_queue.cc:1035] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30957af665ab480d824e86dafe4ef1e8" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:52.992515 30714 consensus_queue.cc:1035] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac7940483ee4ddabe577716373bda35" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:52.997768 30686 mvcc.cc:205] Tried to move back new op lower bound from 7317640368087879680 to 7317640367312445440. Current Snapshot: MvccSnapshot[applied={T|T < 7317640368087879680}]
I20260812 11:14:52.999022 30689 mvcc.cc:205] Tried to move back new op lower bound from 7317640368087879680 to 7317640367312445440. Current Snapshot: MvccSnapshot[applied={T|T < 7317640368087879680}]
I20260812 11:14:53.023701 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:53.018773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:53575 (local address 127.27.109.3:33887)
0812 11:14:53.018977 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:53.018982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:53.019321 (+ 339us) server_negotiation.cc:407] Connection header received
0812 11:14:53.019512 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:53.019516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:53.019596 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:53.019708 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:53.020100 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.020715 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.022223 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.022525 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.022619 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:53.023170 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:53.023275 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:53.023363 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:53.023477 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 11:14:53.023541 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 11:14:53.023823 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:53.018552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33887 (local address 127.27.109.2:53575)
0812 11:14:53.019194 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:53.019213 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:53.019356 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:53.019755 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:53.019758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:53.019782 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:14:53.019944 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:53.019953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.020875 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.020879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:53.022037 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:53.022053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.022673 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.022677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:53.022798 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:53.022921 (+ 123us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:14:53.023381 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:14:53.023393 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 11:14:53.023466 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:14:53.023641 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":204,"threads_started":1}
I20260812 11:14:53.054571 29889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:53.054804 29889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:53.057209 29889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e63d449d4c964e40832f38da2f11db5d: Sending DeleteTablet for 3 replicas of tablet 2ac839017bab4fddb099023a1efa8866
I20260812 11:14:53.057935 30094 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac839017bab4fddb099023a1efa8866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:53 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:44838
I20260812 11:14:53.057984 30520 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac839017bab4fddb099023a1efa8866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:53 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:60584
I20260812 11:14:53.058162 30705 tablet_replica.cc:330] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: stopping tablet replica
I20260812 11:14:53.058346 30705 raft_consensus.cc:2229] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:53.058418 30301 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac839017bab4fddb099023a1efa8866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:53 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:41068
I20260812 11:14:53.058490 30705 raft_consensus.cc:2258] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:53.058725 30706 tablet_replica.cc:330] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: stopping tablet replica
I20260812 11:14:53.058888 30706 raft_consensus.cc:2229] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:14:53.058938 30704 tablet_replica.cc:330] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: stopping tablet replica
I20260812 11:14:53.059105 30704 raft_consensus.cc:2229] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:53.059280 30704 raft_consensus.cc:2258] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:53.059264 30706 raft_consensus.cc:2258] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:53.060150 30705 ts_tablet_manager.cc:1905] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:53.060249 30706 ts_tablet_manager.cc:1905] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:53.060537 30704 ts_tablet_manager.cc:1905] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:53.063261 30706 ts_tablet_manager.cc:1918] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:53.063258 30705 ts_tablet_manager.cc:1918] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:53.063344 30706 log.cc:1196] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/2ac839017bab4fddb099023a1efa8866
I20260812 11:14:53.063344 30705 log.cc:1196] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/2ac839017bab4fddb099023a1efa8866
I20260812 11:14:53.063624 30705 ts_tablet_manager.cc:1939] T 2ac839017bab4fddb099023a1efa8866 P 30957af665ab480d824e86dafe4ef1e8: Deleting consensus metadata
I20260812 11:14:53.063624 30706 ts_tablet_manager.cc:1939] T 2ac839017bab4fddb099023a1efa8866 P f44696484e0d47b8967e142b4f9d3c0c: Deleting consensus metadata
I20260812 11:14:53.064067 30704 ts_tablet_manager.cc:1918] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:53.064147 30704 log.cc:1196] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/2ac839017bab4fddb099023a1efa8866
I20260812 11:14:53.064375 29866 catalog_manager.cc:4841] TS f44696484e0d47b8967e142b4f9d3c0c (127.27.109.3:33887): tablet 2ac839017bab4fddb099023a1efa8866 (table test-table [id=22f18ef0a37043a6b4a6c25b9b2acba6]) successfully deleted
I20260812 11:14:53.064430 30704 ts_tablet_manager.cc:1939] T 2ac839017bab4fddb099023a1efa8866 P bac7940483ee4ddabe577716373bda35: Deleting consensus metadata
I20260812 11:14:53.064728 29864 catalog_manager.cc:4841] TS 30957af665ab480d824e86dafe4ef1e8 (127.27.109.1:41045): tablet 2ac839017bab4fddb099023a1efa8866 (table test-table [id=22f18ef0a37043a6b4a6c25b9b2acba6]) successfully deleted
I20260812 11:14:53.065407 29864 catalog_manager.cc:4841] TS bac7940483ee4ddabe577716373bda35 (127.27.109.2:39541): tablet 2ac839017bab4fddb099023a1efa8866 (table test-table [id=22f18ef0a37043a6b4a6c25b9b2acba6]) successfully deleted
Aug 12 11:14:53 dist-test-slave-7z32 krb5kdc[29819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:14:53.082010 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:53.067199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.27.109.1:41045)
0812 11:14:53.067320 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:53.067326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:53.067897 (+ 571us) server_negotiation.cc:407] Connection header received
0812 11:14:53.068113 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:53.068117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:53.068167 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:53.068220 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:53.069241 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.070061 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.071480 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.071912 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.072088 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:53.076670 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:53.076693 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:53.076720 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:53.076778 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:53.079454 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:53.080108 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:53.080119 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:53.080123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:53.080263 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:53.080617 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:53.080621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:53.080623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:53.081081 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:53.081333 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:53.081594 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 11:14:53.081782 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 11:14:53.083333 30182 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44862
I20260812 11:14:53.097224 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:53.086002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.27.109.62:33537)
0812 11:14:53.086143 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:53.086147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:53.086379 (+ 232us) server_negotiation.cc:407] Connection header received
0812 11:14:53.086530 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:53.086534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:53.086593 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:53.086693 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:53.087678 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.088512 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.089938 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:53.090331 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:53.090441 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:53.092846 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:53.092852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:53.092855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:53.092899 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:53.095045 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:53.095617 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:53.095622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:53.095624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:53.095733 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:53.096076 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:53.096080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:53.096082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:53.096376 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:53.096537 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:53.096841 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 11:14:53.097017 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 11:14:53.098212 29889 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57136
I20260812 11:14:53.100600 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30000
I20260812 11:14:53.180616 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30212
I20260812 11:14:53.199235 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30424
I20260812 11:14:53.220273 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29839
2026-08-12T11:14:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3713 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:14:53 dist-test-slave-7z32 krb5kdc[30779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:14:53 dist-test-slave-7z32 krb5kdc[30779](info): set up 2 sockets
Aug 12 11:14:53 dist-test-slave-7z32 krb5kdc[30779](info): commencing operation
krb5kdc: starting...
W20260812 11:14:54.619659 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:14:54 dist-test-slave-7z32 krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:14:54.812745 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46505
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:46505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:54.952288 30795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:54.952641 30795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:54.952687 30795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:54.957558 30795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:14:54.957670 30795 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:14:54.957690 30795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:54.957710 30795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:54.957726 30795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:14:54.957741 30795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:14:54.965834 30795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:46505
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:54.967274 30795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:54.968472 30795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:54.981139 30806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.981379 30802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.981139 30803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:54.982188 30795 server_base.cc:1034] running on GCE node
I20260812 11:14:54.983263 30795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:54.984869 30795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:54.988389 30795 hybrid_clock.cc:648] HybridClock initialized: now 1786533294988298 us; error 95 us; skew 500 ppm
Aug 12 11:14:54 dist-test-slave-7z32 krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:54.992791 30795 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:14:54.994138 30795 webserver.cc:460] Webserver started at http://127.27.109.62:37555/ using document root <none> and password file <none>
I20260812 11:14:54.995011 30795 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:54.995128 30795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:54.995402 30795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:54.997934 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "837038e80c7545c1ba933a5308c9be64"
format_stamp: "Formatted at 2026-08-12 11:14:54 on dist-test-slave-7z32"
server_key: "9932bbb7393ff60f63ac54a3822da2ac"
server_key_iv: "5a43779a31242e6943622f0073baee56"
server_key_version: "encryptionkey@0"
I20260812 11:14:54.998693 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "837038e80c7545c1ba933a5308c9be64"
format_stamp: "Formatted at 2026-08-12 11:14:54 on dist-test-slave-7z32"
server_key: "9932bbb7393ff60f63ac54a3822da2ac"
server_key_iv: "5a43779a31242e6943622f0073baee56"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.004360 30795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:14:55.008445 30817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:55.010502 30795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260812 11:14:55.010749 30795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "837038e80c7545c1ba933a5308c9be64"
format_stamp: "Formatted at 2026-08-12 11:14:54 on dist-test-slave-7z32"
server_key: "9932bbb7393ff60f63ac54a3822da2ac"
server_key_iv: "5a43779a31242e6943622f0073baee56"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.010990 30795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:55.040126 30795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:55.044006 30795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:55.044245 30795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:55.054423 30795 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:46505
I20260812 11:14:55.054479 30918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:46505 every 8 connection(s)
I20260812 11:14:55.055703 30795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:14:55.059846 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:55.064677 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30795
I20260812 11:14:55.064854 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:14:55.065197 28084 external_mini_cluster.cc:1428] Setting key b318919d1315dc2549867e89a8078886
I20260812 11:14:55.068480 30920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Bootstrap starting.
I20260812 11:14:55.071827 30920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:55.072851 30920 log.cc:824] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:55.076303 30920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: No bootstrap required, opened a new log
Aug 12 11:14:55 dist-test-slave-7z32 krb5kdc[30779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:55.081422 30920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837038e80c7545c1ba933a5308c9be64" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46505 } }
I20260812 11:14:55.081844 30920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:55.081950 30920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837038e80c7545c1ba933a5308c9be64, State: Initialized, Role: FOLLOWER
I20260812 11:14:55.082697 30920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837038e80c7545c1ba933a5308c9be64" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46505 } }
I20260812 11:14:55.082980 30920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:14:55.083063 30920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:14:55.083204 30920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:55.084302 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.067396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.27.109.62:46505)
0812 11:14:55.068095 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:55.068118 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:55.068336 (+ 218us) server_negotiation.cc:407] Connection header received
0812 11:14:55.069456 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:55.069480 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:55.069929 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:55.070295 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:55.071435 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.072694 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.073903 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.074492 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.074633 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.078194 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:55.078201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:55.078226 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:55.078278 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:55.080347 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.080894 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.080900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.080907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.081044 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.081375 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.081378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.081380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.081926 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:55.082175 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:55.082748 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 11:14:55.083322 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":179,"threads_started":1}
I20260812 11:14:55.084856 30920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837038e80c7545c1ba933a5308c9be64" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46505 } }
I20260812 11:14:55.085350 30920 leader_election.cc:302] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 837038e80c7545c1ba933a5308c9be64; no voters:
I20260812 11:14:55.085850 30920 leader_election.cc:288] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:14:55.085954 30928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:55.086171 30928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [term 1 LEADER]: Becoming Leader. State: Replica: 837038e80c7545c1ba933a5308c9be64, State: Running, Role: LEADER
I20260812 11:14:55.086585 30928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837038e80c7545c1ba933a5308c9be64" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46505 } }
I20260812 11:14:55.087656 30920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:14:55.088970 30931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 837038e80c7545c1ba933a5308c9be64. Latest consensus state: current_term: 1 leader_uuid: "837038e80c7545c1ba933a5308c9be64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837038e80c7545c1ba933a5308c9be64" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46505 } } }
I20260812 11:14:55.088938 30929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "837038e80c7545c1ba933a5308c9be64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837038e80c7545c1ba933a5308c9be64" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46505 } } }
I20260812 11:14:55.089203 30931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:55.089242 30929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64 [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:55.089730 30942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:14:55.093401 30942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:14:55.101130 30942 catalog_manager.cc:1347] Generated new cluster ID: 3d26128480df44dbae6eb3bab7a1f82d
I20260812 11:14:55.101257 30942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:14:55.126408 30942 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:14:55.127601 30942 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:14:55.133504 30942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Generated new TSK 0
I20260812 11:14:55.134793 30942 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:14:55.184757 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46505
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:55.320427 30955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:55.320811 30955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:55.320905 30955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:55.325664 30955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:55.325824 30955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:55.325914 30955 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:14:55.325969 30955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:14:55.326050 30955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:14:55.331918 30955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:55.333205 30955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:55.334362 30955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:55.345760 30962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:55.345854 30963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:55.345854 30965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:55.346385 30955 server_base.cc:1034] running on GCE node
I20260812 11:14:55.347149 30955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:55.347996 30955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:55.349255 30955 hybrid_clock.cc:648] HybridClock initialized: now 1786533295349225 us; error 55 us; skew 500 ppm
Aug 12 11:14:55 dist-test-slave-7z32 krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:55.353461 30955 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:14:55.354712 30955 webserver.cc:460] Webserver started at http://127.27.109.1:43767/ using document root <none> and password file <none>
I20260812 11:14:55.355419 30955 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:55.355532 30955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:55.355806 30955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:55.358165 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "07037b345dfc454eb0514632d807681d"
format_stamp: "Formatted at 2026-08-12 11:14:55 on dist-test-slave-7z32"
server_key: "b7a7fce24dc8a80a32b178a7fa95ec7b"
server_key_iv: "ea90bd346e78fcc58c4d3702bb6d7034"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.358934 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "07037b345dfc454eb0514632d807681d"
format_stamp: "Formatted at 2026-08-12 11:14:55 on dist-test-slave-7z32"
server_key: "b7a7fce24dc8a80a32b178a7fa95ec7b"
server_key_iv: "ea90bd346e78fcc58c4d3702bb6d7034"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.364588 30955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:14:55.368453 30973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:55.370290 30955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:14:55.370512 30955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "07037b345dfc454eb0514632d807681d"
format_stamp: "Formatted at 2026-08-12 11:14:55 on dist-test-slave-7z32"
server_key: "b7a7fce24dc8a80a32b178a7fa95ec7b"
server_key_iv: "ea90bd346e78fcc58c4d3702bb6d7034"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.370714 30955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:55.412948 30955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:55.417541 30955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:55.417788 30955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:55.419663 30955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:55.419746 30955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:55.419827 30955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:55.419872 30955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:55.432695 30955 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:32895
I20260812 11:14:55.433012 31138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:32895 every 8 connection(s)
I20260812 11:14:55.434051 30955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
Aug 12 11:14:55 dist-test-slave-7z32 krb5kdc[30779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:55.437654 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30955
I20260812 11:14:55.437841 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:14:55.438239 28084 external_mini_cluster.cc:1428] Setting key 9d8dd6c867e28220189b528dd0bfc651
I20260812 11:14:55.440585 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.421336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:49245 (local address 127.27.109.62:46505)
0812 11:14:55.421525 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:55.421530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:55.423285 (+ 1755us) server_negotiation.cc:407] Connection header received
0812 11:14:55.424772 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:55.424776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:55.424857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:55.424955 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:55.427318 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.428422 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.430148 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.430463 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.430603 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.435394 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:55.435399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:55.435402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:55.435435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:55.437070 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.438115 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.438119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.438121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.438247 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.438818 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.438874 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.438879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.439182 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:55.439305 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:55.440181 (+ 876us) server_negotiation.cc:299] Negotiation successful
0812 11:14:55.440367 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:14:55.441929 30984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.421816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46505 (local address 127.27.109.1:49245)
0812 11:14:55.423012 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.423101 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:55.424463 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:55.425311 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:55.425326 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:55.425959 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:55.426981 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.427013 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.428645 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.428652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.429933 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.429952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.430702 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.430707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.431027 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.432089 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:55.432113 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:55.435107 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:55.437283 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.437292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.437316 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.437833 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.438419 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.438423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.438425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.438673 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.439320 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:55.439327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:55.439806 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 11:14:55.440073 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 11:14:55.440618 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":167,"threads_started":1}
I20260812 11:14:55.444399 31139 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46505
I20260812 11:14:55.444630 31139 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:55.445407 31139 heartbeater.cc:511] Master 127.27.109.62:46505 requested a full tablet report, sending...
I20260812 11:14:55.447990 30839 ts_manager.cc:194] Registered new tserver with Master: 07037b345dfc454eb0514632d807681d (127.27.109.1:32895)
I20260812 11:14:55.450218 30839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:49245
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:14:55.458657 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.447409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46505 (local address 127.27.109.1:50597)
0812 11:14:55.447821 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.447837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:55.447971 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:55.448412 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:55.448414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:55.448644 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:55.448764 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.448770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.449744 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.449750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.450517 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.450533 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.451206 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.451210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.451322 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.451982 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:55.451995 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:55.452551 (+ 556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:55.455817 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.455821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.455824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.456211 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.457072 (+ 861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.457075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.457077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.457185 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.457974 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:55.457977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:55.458103 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 11:14:55.458292 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 11:14:55.458463 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":121,"threads_started":1}
I20260812 11:14:55.458897 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.447452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50597 (local address 127.27.109.62:46505)
0812 11:14:55.447620 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:55.447626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:55.447920 (+ 294us) server_negotiation.cc:407] Connection header received
0812 11:14:55.448089 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:55.448092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:55.448179 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:55.448261 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:55.448903 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.449599 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.450670 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.451069 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.451197 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.452797 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:55.452803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:55.452807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:55.452848 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:55.455539 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.456520 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.456524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.456526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.456841 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.457330 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.457333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.457335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.457721 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:55.457922 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:55.458381 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 11:14:55.458575 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:14:55.484481 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46505
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:55.656874 31153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:55.657377 31153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:55.657514 31153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:55.663223 31153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:55.663343 31153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:55.663400 31153 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:14:55.663419 31153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:14:55.663501 31153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:14:55.670725 31153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:55.672712 31153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:55.674566 31153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:55.690196 31162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:55.690253 31161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:55.690213 31164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:55.690858 31153 server_base.cc:1034] running on GCE node
I20260812 11:14:55.691501 31153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:55.692485 31153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:55.693724 31153 hybrid_clock.cc:648] HybridClock initialized: now 1786533295693681 us; error 56 us; skew 500 ppm
Aug 12 11:14:55 dist-test-slave-7z32 krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:55.699178 31153 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:14:55.701152 31153 webserver.cc:460] Webserver started at http://127.27.109.2:46183/ using document root <none> and password file <none>
I20260812 11:14:55.702515 31153 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:55.702665 31153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:55.703135 31153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:55.706941 31153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "d0495a0f57ca472ca094269eb943d46a"
format_stamp: "Formatted at 2026-08-12 11:14:55 on dist-test-slave-7z32"
server_key: "9daea6c6998850f3409b4017e65cfb08"
server_key_iv: "79065db3514117857d468791cfaf69fa"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.707950 31153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "d0495a0f57ca472ca094269eb943d46a"
format_stamp: "Formatted at 2026-08-12 11:14:55 on dist-test-slave-7z32"
server_key: "9daea6c6998850f3409b4017e65cfb08"
server_key_iv: "79065db3514117857d468791cfaf69fa"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.713948 31153 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 11:14:55.718559 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:55.720786 31153 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:14:55.721091 31153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "d0495a0f57ca472ca094269eb943d46a"
format_stamp: "Formatted at 2026-08-12 11:14:55 on dist-test-slave-7z32"
server_key: "9daea6c6998850f3409b4017e65cfb08"
server_key_iv: "79065db3514117857d468791cfaf69fa"
server_key_version: "encryptionkey@0"
I20260812 11:14:55.721312 31153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:55.757110 31153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:55.762566 31153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:55.762941 31153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:55.765178 31153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:55.765283 31153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:55.765362 31153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:55.765422 31153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:14:55 dist-test-slave-7z32 krb5kdc[30779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:55.781675 31153 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44915
I20260812 11:14:55.781791 31351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44915 every 8 connection(s)
I20260812 11:14:55.783396 31153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:14:55.786242 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.766767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:53249 (local address 127.27.109.62:46505)
0812 11:14:55.766962 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:55.766968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:55.768789 (+ 1821us) server_negotiation.cc:407] Connection header received
0812 11:14:55.770449 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:55.770453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:55.770523 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:55.770627 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:55.773320 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.774187 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.776078 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.776471 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.776615 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.781103 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:55.781107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:55.781110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:55.781142 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:55.783357 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.784179 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.784184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.784186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.784315 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.784794 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.784799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.784801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.785126 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:55.785242 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:55.785892 (+ 650us) server_negotiation.cc:299] Negotiation successful
0812 11:14:55.786045 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:14:55.787238 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.767665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46505 (local address 127.27.109.2:53249)
0812 11:14:55.768520 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.768615 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:55.770156 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:55.771007 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:55.771021 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:55.771823 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:55.772953 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.772993 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.774381 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.774386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.775890 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.775905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.776751 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.776758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.777127 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.778174 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:55.778211 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:55.780821 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:55.783581 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.783590 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.783612 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.783982 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.784437 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.784440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.784442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.784647 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.785314 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:55.785320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:55.785620 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 11:14:55.785808 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 11:14:55.786142 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":205,"threads_started":1}
I20260812 11:14:55.789139 31352 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46505
I20260812 11:14:55.789415 31352 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:55.789433 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31153
I20260812 11:14:55.789573 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:14:55.789954 28084 external_mini_cluster.cc:1428] Setting key b7848cecb3a27ad96ab16a3dcc76d122
I20260812 11:14:55.790297 31352 heartbeater.cc:511] Master 127.27.109.62:46505 requested a full tablet report, sending...
I20260812 11:14:55.792091 30839 ts_manager.cc:194] Registered new tserver with Master: d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915)
I20260812 11:14:55.793149 30839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:53249
I20260812 11:14:55.802627 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.792568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:49573 (local address 127.27.109.62:46505)
0812 11:14:55.792711 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:55.792717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:55.793009 (+ 292us) server_negotiation.cc:407] Connection header received
0812 11:14:55.793183 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:55.793186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:55.793235 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:55.793335 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:55.793998 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.795107 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.796299 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.796588 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.796720 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.798432 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:55.798436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:55.798439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:55.798476 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:55.800594 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.801089 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.801093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.801095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.801182 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.801504 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.801507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.801509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.801774 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:55.801896 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:55.802231 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 11:14:55.802401 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:14:55.803776 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.792493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46505 (local address 127.27.109.2:49573)
0812 11:14:55.792904 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.792918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:55.793067 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:55.793471 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:55.793474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:55.793694 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:55.793824 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.793830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.795343 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.795355 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.796140 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.796152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.796740 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.796745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.796942 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.797614 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:55.797627 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:55.798190 (+ 563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:55.800735 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.800738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.800740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.800970 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.801279 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.801283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.801285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.801392 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.801883 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:55.801886 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:55.802006 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 11:14:55.802934 (+ 928us) client_negotiation.cc:240] Negotiation successful
0812 11:14:55.803402 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":7,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:14:55.836728 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46505
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:56.018194 31367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:56.019356 31367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:56.019464 31367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:56.032655 31367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:56.032831 31367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:56.032900 31367 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:14:56.032917 31367 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:14:56.032958 31367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:14:56.041903 31367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:14:56.043900 31367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:56.045619 31367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:56.062630 31377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:56.063215 31375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:56.063321 31383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:56.064625 31367 server_base.cc:1034] running on GCE node
I20260812 11:14:56.065392 31367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:56.066387 31367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:56.069744 31367 hybrid_clock.cc:648] HybridClock initialized: now 1786533296069688 us; error 80 us; skew 500 ppm
Aug 12 11:14:56 dist-test-slave-7z32 krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:56.074774 31367 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:14:56.076484 31367 webserver.cc:460] Webserver started at http://127.27.109.3:43357/ using document root <none> and password file <none>
I20260812 11:14:56.077275 31367 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:56.077373 31367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:56.077649 31367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:56.080968 31367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "2201be271dff4291afe54af4360114cd"
format_stamp: "Formatted at 2026-08-12 11:14:56 on dist-test-slave-7z32"
server_key: "5e5c982219a95d5c471e82c3b0d9f3bb"
server_key_iv: "87b698a1a07b7ce389109abbdba99dad"
server_key_version: "encryptionkey@0"
I20260812 11:14:56.082024 31367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "2201be271dff4291afe54af4360114cd"
format_stamp: "Formatted at 2026-08-12 11:14:56 on dist-test-slave-7z32"
server_key: "5e5c982219a95d5c471e82c3b0d9f3bb"
server_key_iv: "87b698a1a07b7ce389109abbdba99dad"
server_key_version: "encryptionkey@0"
I20260812 11:14:56.088320 31367 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:14:56.092876 31393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:56.095827 31367 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:14:56.096036 31367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "2201be271dff4291afe54af4360114cd"
format_stamp: "Formatted at 2026-08-12 11:14:56 on dist-test-slave-7z32"
server_key: "5e5c982219a95d5c471e82c3b0d9f3bb"
server_key_iv: "87b698a1a07b7ce389109abbdba99dad"
server_key_version: "encryptionkey@0"
I20260812 11:14:56.096228 31367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:56.123723 31367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:56.128615 31367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:56.128885 31367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:56.130707 31367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:56.130769 31367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:56.130813 31367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:56.130935 31367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 12 11:14:56 dist-test-slave-7z32 krb5kdc[30779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:56.146154 31367 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:42839
I20260812 11:14:56.146291 31559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:42839 every 8 connection(s)
I20260812 11:14:56.147624 31367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:14:56.153026 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.132425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:52555 (local address 127.27.109.62:46505)
0812 11:14:56.132665 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.132670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.134213 (+ 1543us) server_negotiation.cc:407] Connection header received
0812 11:14:56.136128 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.136133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.136222 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.136357 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:56.138873 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.139849 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.141880 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.142183 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.142342 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.146987 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:56.146992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:56.146999 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:56.147047 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:56.149501 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.150388 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.150396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.150398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.150541 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.151355 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.151359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.151361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.151663 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:56.151811 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.152607 (+ 796us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.152817 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:14:56.153873 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31367
I20260812 11:14:56.154067 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:14:56.154399 28084 external_mini_cluster.cc:1428] Setting key 7476b208338377766d34a8e99af3d991
I20260812 11:14:56.154619 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.132968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46505 (local address 127.27.109.3:52555)
0812 11:14:56.133921 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.134026 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.135767 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.136679 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.136697 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.137464 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:56.138583 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.138612 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.140033 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.140037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.141666 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.141680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.142419 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.142429 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.142722 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.144138 (+ 1416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:56.144169 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:56.146637 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:56.149737 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:56.149748 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:56.149775 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:56.150219 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:56.150879 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:56.150890 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:56.150893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:56.151193 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:56.151811 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:56.151832 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:56.152262 (+ 430us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.152542 (+ 280us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.153153 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":170,"threads_started":1}
I20260812 11:14:56.157342 31560 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46505
I20260812 11:14:56.157612 31560 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:56.158603 31560 heartbeater.cc:511] Master 127.27.109.62:46505 requested a full tablet report, sending...
I20260812 11:14:56.160636 30839 ts_manager.cc:194] Registered new tserver with Master: 2201be271dff4291afe54af4360114cd (127.27.109.3:42839)
I20260812 11:14:56.161952 30839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:52555
I20260812 11:14:56.171176 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:14:56.171782 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.160129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46505 (local address 127.27.109.3:48539)
0812 11:14:56.160709 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.160724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.160941 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.161524 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.161526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.161789 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:56.161936 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.161943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.163501 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.163504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.164463 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.164476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.165124 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.165127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.165231 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.165899 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:56.165912 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:56.166313 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:56.169235 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:56.169242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:56.169246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:56.169661 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:56.170163 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:56.170169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:56.170171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:56.170295 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:56.171020 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:56.171023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:56.171179 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.171326 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.171561 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":11,"thread_start_us":291,"threads_started":1}
I20260812 11:14:56.171875 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.160115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48539 (local address 127.27.109.62:46505)
0812 11:14:56.160337 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.160340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.160928 (+ 588us) server_negotiation.cc:407] Connection header received
0812 11:14:56.161256 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.161260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.161341 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.161456 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:56.162205 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.163169 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.164686 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.164949 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.165082 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.166528 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:56.166532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:56.166536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:56.166703 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:56.168979 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.169834 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.169839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.169841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.170026 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.170420 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.170424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.170426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.170881 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:56.171009 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.171488 (+ 479us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.171670 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":67}
Aug 12 11:14:56 dist-test-slave-7z32 krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:14:56 dist-test-slave-7z32 krb5kdc[30779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:56.214342 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.200035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.27.109.62:46505)
0812 11:14:56.200379 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.200384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.200408 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:14:56.200656 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.200660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.200743 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.200982 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:56.201854 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.202569 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.203649 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.203951 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.204086 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.207278 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:56.207284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:56.207287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:56.207341 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:56.209784 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.210713 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.210717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.210719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.211559 (+ 840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.212094 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.212099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.212100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.212635 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:56.213233 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.213882 (+ 649us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.214148 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260812 11:14:56.220781 30839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:56.225414 30839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:56.248135 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.240156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42839 (local address 127.0.0.1:49820)
0812 11:14:56.240745 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.240819 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.240969 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.241787 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.241795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.241823 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:56.242062 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.242071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.243894 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.243897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.245761 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.245780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.247474 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.247478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.247594 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.247628 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.247720 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.247825 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":150,"thread_start_us":246,"threads_started":1}
I20260812 11:14:56.248006 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.240700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:32895 (local address 127.0.0.1:40278)
0812 11:14:56.241390 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.241406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.241525 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.242406 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.242409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.242421 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:56.242678 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.242686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.244193 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.244197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.245443 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.245454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.247191 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.247195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.247313 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.247351 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.247523 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.247733 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":166,"thread_start_us":148,"threads_started":1}
I20260812 11:14:56.248484 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.240347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.27.109.3:42839)
0812 11:14:56.241066 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.241084 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.241119 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:14:56.241233 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.241239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.241561 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.242245 (+ 684us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:56.242424 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.243754 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.245942 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.246784 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.246920 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.247121 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.248133 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.248307 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":169,"threads_started":1}
I20260812 11:14:56.249171 31589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.241162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44915 (local address 127.0.0.1:42854)
0812 11:14:56.241752 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.241768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.241884 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.242377 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.242383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.242405 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:56.242647 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.242657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.244230 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.244239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.245760 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.245773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.248549 (+ 2776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.248555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.248659 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.248745 (+ 86us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.248900 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.249008 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":184,"spinlock_wait_cycles":10240,"thread_start_us":139,"threads_started":1}
I20260812 11:14:56.249236 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.241357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.27.109.2:44915)
0812 11:14:56.241854 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.241873 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.241899 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:14:56.241987 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.241992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.242203 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.242358 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:56.242846 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.244101 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.246943 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.247868 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.247977 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.248175 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.248953 (+ 778us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.249064 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":94,"threads_started":1}
I20260812 11:14:56.249809 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.241532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.27.109.1:32895)
0812 11:14:56.241932 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.241942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.241969 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:14:56.242057 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.242061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.242228 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.242618 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:56.242803 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.244051 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.245609 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.246684 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.246873 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.247103 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.247936 (+ 833us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.248107 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":103,"threads_started":1}
I20260812 11:14:56.252343 31461 tablet_service.cc:1467] Processing CreateTablet for tablet 773696bb5665486da5e4041022ea99f0 (DEFAULT_TABLE table=test-table [id=94f93cd7ef1d48349bde6a13c0310ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:56.253079 31049 tablet_service.cc:1467] Processing CreateTablet for tablet 773696bb5665486da5e4041022ea99f0 (DEFAULT_TABLE table=test-table [id=94f93cd7ef1d48349bde6a13c0310ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:56.253073 31246 tablet_service.cc:1467] Processing CreateTablet for tablet 773696bb5665486da5e4041022ea99f0 (DEFAULT_TABLE table=test-table [id=94f93cd7ef1d48349bde6a13c0310ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:56.254350 31461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773696bb5665486da5e4041022ea99f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.254505 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773696bb5665486da5e4041022ea99f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.254505 31246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773696bb5665486da5e4041022ea99f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.264199 31592 tablet_bootstrap.cc:492] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Bootstrap starting.
I20260812 11:14:56.266316 31593 tablet_bootstrap.cc:492] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Bootstrap starting.
I20260812 11:14:56.267375 31595 tablet_bootstrap.cc:492] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Bootstrap starting.
I20260812 11:14:56.268711 31593 tablet_bootstrap.cc:654] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.269071 31592 tablet_bootstrap.cc:654] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.269702 31593 log.cc:824] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:56.270164 31592 log.cc:824] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:56.270293 31595 tablet_bootstrap.cc:654] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.271343 31595 log.cc:824] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:56.272481 31593 tablet_bootstrap.cc:492] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: No bootstrap required, opened a new log
I20260812 11:14:56.272768 31593 ts_tablet_manager.cc:1397] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 11:14:56.273049 31592 tablet_bootstrap.cc:492] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: No bootstrap required, opened a new log
I20260812 11:14:56.274129 31592 ts_tablet_manager.cc:1397] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260812 11:14:56.274294 31595 tablet_bootstrap.cc:492] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: No bootstrap required, opened a new log
I20260812 11:14:56.274583 31595 ts_tablet_manager.cc:1397] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:14:56.277511 31593 raft_consensus.cc:348] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.277832 31593 raft_consensus.cc:374] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.277890 31593 raft_consensus.cc:729] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07037b345dfc454eb0514632d807681d, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.278395 31593 consensus_queue.cc:260] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.279299 31139 heartbeater.cc:503] Master 127.27.109.62:46505 was elected leader, sending a full tablet report...
I20260812 11:14:56.279670 31592 raft_consensus.cc:348] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.279670 31595 raft_consensus.cc:348] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.280112 31595 raft_consensus.cc:374] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.280115 31592 raft_consensus.cc:374] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.280186 31592 raft_consensus.cc:729] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2201be271dff4291afe54af4360114cd, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.280206 31595 raft_consensus.cc:729] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0495a0f57ca472ca094269eb943d46a, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.280853 31595 consensus_queue.cc:260] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.281153 31593 ts_tablet_manager.cc:1428] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260812 11:14:56.281834 31352 heartbeater.cc:503] Master 127.27.109.62:46505 was elected leader, sending a full tablet report...
I20260812 11:14:56.281749 31592 consensus_queue.cc:260] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.283255 31592 ts_tablet_manager.cc:1428] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260812 11:14:56.283676 31560 heartbeater.cc:503] Master 127.27.109.62:46505 was elected leader, sending a full tablet report...
I20260812 11:14:56.284339 31603 raft_consensus.cc:482] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:56.284577 31603 raft_consensus.cc:504] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.285768 31595 ts_tablet_manager.cc:1428] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260812 11:14:56.288020 31603 leader_election.cc:288] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915), 07037b345dfc454eb0514632d807681d (127.27.109.1:32895)
I20260812 11:14:56.292073 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.287445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:60385 (local address 127.27.109.2:44915)
0812 11:14:56.287796 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.287801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.287833 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:14:56.287893 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.287899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.287968 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.288065 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:56.288573 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.289271 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.290628 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.291307 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.291452 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.291505 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.291843 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.291944 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260812 11:14:56.292075 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.286715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44915 (local address 127.27.109.3:60385)
0812 11:14:56.287082 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.287102 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.287204 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.288111 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.288117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.288144 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:56.288350 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.288357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.289415 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.289420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.290466 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.290478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.291468 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.291471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.291685 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.291723 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.291792 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.291870 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"spinlock_wait_cycles":32640}
I20260812 11:14:56.292994 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773696bb5665486da5e4041022ea99f0" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a" is_pre_election: true
I20260812 11:14:56.293398 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.288030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38643 (local address 127.27.109.1:32895)
0812 11:14:56.288162 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.288168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.288185 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:14:56.288348 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.288351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.288397 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.288487 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:56.289608 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.290250 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.291980 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.292694 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.292840 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.292899 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.293171 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.293270 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:14:56.293430 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.287714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:32895 (local address 127.27.109.3:38643)
0812 11:14:56.288105 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.288119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.288216 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.288710 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.288714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.288808 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:56.289465 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.289477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.290366 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.290371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.291803 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.291818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.292899 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.292903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.293012 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.293041 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.293125 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.293194 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":151,"thread_start_us":181,"threads_started":1}
I20260812 11:14:56.293498 31284 raft_consensus.cc:2454] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 0.
I20260812 11:14:56.294278 31399 leader_election.cc:302] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2201be271dff4291afe54af4360114cd, d0495a0f57ca472ca094269eb943d46a; no voters:
I20260812 11:14:56.294589 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773696bb5665486da5e4041022ea99f0" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07037b345dfc454eb0514632d807681d" is_pre_election: true
I20260812 11:14:56.294740 31603 raft_consensus.cc:2781] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:56.294886 31603 raft_consensus.cc:482] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:56.294943 31603 raft_consensus.cc:3037] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.295004 31074 raft_consensus.cc:2454] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 0.
I20260812 11:14:56.296550 31603 raft_consensus.cc:504] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.297068 31603 leader_election.cc:288] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 election: Requested vote from peers d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915), 07037b345dfc454eb0514632d807681d (127.27.109.1:32895)
I20260812 11:14:56.297524 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773696bb5665486da5e4041022ea99f0" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a"
I20260812 11:14:56.297724 31284 raft_consensus.cc:3037] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.299098 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773696bb5665486da5e4041022ea99f0" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07037b345dfc454eb0514632d807681d"
I20260812 11:14:56.299310 31074 raft_consensus.cc:3037] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.299373 31284 raft_consensus.cc:2454] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 1.
I20260812 11:14:56.300305 31399 leader_election.cc:302] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2201be271dff4291afe54af4360114cd, d0495a0f57ca472ca094269eb943d46a; no voters:
I20260812 11:14:56.300509 31603 raft_consensus.cc:2781] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:56.300665 31603 raft_consensus.cc:686] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 1 LEADER]: Becoming Leader. State: Replica: 2201be271dff4291afe54af4360114cd, State: Running, Role: LEADER
I20260812 11:14:56.301084 31074 raft_consensus.cc:2454] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 1.
I20260812 11:14:56.301188 31603 consensus_queue.cc:237] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } }
I20260812 11:14:56.305785 30843 catalog_manager.cc:5495] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2201be271dff4291afe54af4360114cd (127.27.109.3). New cstate: current_term: 1 leader_uuid: "2201be271dff4291afe54af4360114cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } health_report { overall_health: UNKNOWN } } }
Aug 12 11:14:56 dist-test-slave-7z32 krb5kdc[30779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:56.362072 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.349489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46505 (local address 127.0.0.1:46088)
0812 11:14:56.350201 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.350219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.350519 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.350860 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.350863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.351415 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:56.351518 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.351526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.352596 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.352603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.353223 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.353232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.353737 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.353741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.353828 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.354600 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:56.354614 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:56.357189 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:56.359704 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:56.359727 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:56.359737 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:56.360165 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:56.360666 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:56.360669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:56.360671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:56.360780 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:56.361343 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:56.361353 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:56.361539 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.361675 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.361882 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":383,"threads_started":1}
I20260812 11:14:56.362172 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.349483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.27.109.62:46505)
0812 11:14:56.349768 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.349773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.350421 (+ 648us) server_negotiation.cc:407] Connection header received
0812 11:14:56.350663 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.350666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.350706 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.350809 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:56.351771 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.352448 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.353369 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.353601 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.353729 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.357448 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:56.357453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:56.357455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:56.357489 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:56.359502 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.360353 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.360357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.360359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.360472 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:56.360934 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:56.360937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:56.360938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:56.361192 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:56.361292 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.361755 (+ 463us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.362016 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":169}
I20260812 11:14:56.368432 30843 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:46088:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:14:56.369609 30843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:56.376760 31049 tablet_service.cc:1467] Processing CreateTablet for tablet f206a5477b694c20acba87ef049b4750 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8051c2525edd44d98bad543a9e542fc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:14:56.376948 31461 tablet_service.cc:1467] Processing CreateTablet for tablet f206a5477b694c20acba87ef049b4750 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8051c2525edd44d98bad543a9e542fc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:14:56.376925 31246 tablet_service.cc:1467] Processing CreateTablet for tablet f206a5477b694c20acba87ef049b4750 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8051c2525edd44d98bad543a9e542fc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:14:56.377581 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f206a5477b694c20acba87ef049b4750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.377876 31461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f206a5477b694c20acba87ef049b4750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.377876 31246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f206a5477b694c20acba87ef049b4750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.380833 31593 tablet_bootstrap.cc:492] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: Bootstrap starting.
I20260812 11:14:56.381898 31593 tablet_bootstrap.cc:654] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.382073 31595 tablet_bootstrap.cc:492] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: Bootstrap starting.
I20260812 11:14:56.382787 31592 tablet_bootstrap.cc:492] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: Bootstrap starting.
I20260812 11:14:56.383419 31595 tablet_bootstrap.cc:654] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.383774 31593 tablet_bootstrap.cc:492] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: No bootstrap required, opened a new log
I20260812 11:14:56.383888 31593 ts_tablet_manager.cc:1397] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:56.384258 31592 tablet_bootstrap.cc:654] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.384635 31593 raft_consensus.cc:348] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.384814 31593 raft_consensus.cc:374] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.384791 31595 tablet_bootstrap.cc:492] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: No bootstrap required, opened a new log
I20260812 11:14:56.384853 31593 raft_consensus.cc:729] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07037b345dfc454eb0514632d807681d, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.384912 31595 ts_tablet_manager.cc:1397] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:56.385113 31593 consensus_queue.cc:260] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.385463 31595 raft_consensus.cc:348] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.385492 31601 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.385619 31595 raft_consensus.cc:374] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.385655 31595 raft_consensus.cc:729] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0495a0f57ca472ca094269eb943d46a, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.385669 31601 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:56.385802 31595 consensus_queue.cc:260] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.385947 31593 ts_tablet_manager.cc:1428] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:56.386060 31592 tablet_bootstrap.cc:492] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: No bootstrap required, opened a new log
I20260812 11:14:56.386173 31592 ts_tablet_manager.cc:1397] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260812 11:14:56.386118 31602 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.386267 31602 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:56.386521 31595 ts_tablet_manager.cc:1428] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:56.386725 31592 raft_consensus.cc:348] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.386878 31592 raft_consensus.cc:374] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.386914 31592 raft_consensus.cc:729] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2201be271dff4291afe54af4360114cd, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.387081 31592 consensus_queue.cc:260] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.387394 31592 ts_tablet_manager.cc:1428] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:14:56.387466 31603 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.387609 31603 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 11:14:56.399248 31561 tablet.cc:2369] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:14:56.436311 31143 tablet.cc:2369] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:14:56.439606 31601 raft_consensus.cc:482] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:56.439786 31601 raft_consensus.cc:504] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.441160 31601 leader_election.cc:288] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915), 2201be271dff4291afe54af4360114cd (127.27.109.3:42839)
I20260812 11:14:56.447528 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.442905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43909 (local address 127.27.109.3:42839)
0812 11:14:56.443087 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.443091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.443106 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:14:56.443236 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.443239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.443289 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.443374 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:56.443653 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.444494 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.446002 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.446676 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.446929 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.446973 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.447289 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.447380 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:14:56.447543 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.441784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:39803 (local address 127.27.109.2:44915)
0812 11:14:56.442012 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.442020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.442456 (+ 436us) server_negotiation.cc:407] Connection header received
0812 11:14:56.442642 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.442646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.442708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.442882 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:56.443405 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.444218 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.446003 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.446714 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.446908 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.446970 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.447347 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.447421 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 11:14:56.447681 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.442488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42839 (local address 127.27.109.1:43909)
0812 11:14:56.443008 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.443022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.443110 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.443435 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.443437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.443457 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:56.443539 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.443544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.444659 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.444662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.445849 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.445856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.446930 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.446933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.447097 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.447138 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.447241 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.447434 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":90,"thread_start_us":149,"threads_started":1}
I20260812 11:14:56.447897 31626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.441812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44915 (local address 127.27.109.1:39803)
0812 11:14:56.442314 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.442332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.442497 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.443039 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.443047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.443075 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:56.443263 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.443269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.444346 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.444350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.445838 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.445852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.446915 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.446919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.447142 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.447159 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.447241 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.447431 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":102,"thread_start_us":232,"threads_started":1}
I20260812 11:14:56.448525 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f206a5477b694c20acba87ef049b4750" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a" is_pre_election: true
I20260812 11:14:56.448547 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f206a5477b694c20acba87ef049b4750" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2201be271dff4291afe54af4360114cd" is_pre_election: true
I20260812 11:14:56.448733 31284 raft_consensus.cc:2454] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 0.
I20260812 11:14:56.448791 31493 raft_consensus.cc:2454] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 0.
I20260812 11:14:56.449286 30976 leader_election.cc:302] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07037b345dfc454eb0514632d807681d, d0495a0f57ca472ca094269eb943d46a; no voters:
I20260812 11:14:56.449734 31601 raft_consensus.cc:2781] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:56.449851 31601 raft_consensus.cc:482] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:56.449930 31601 raft_consensus.cc:3037] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.451210 31601 raft_consensus.cc:504] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.451674 31601 leader_election.cc:288] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 election: Requested vote from peers d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915), 2201be271dff4291afe54af4360114cd (127.27.109.3:42839)
I20260812 11:14:56.452462 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f206a5477b694c20acba87ef049b4750" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2201be271dff4291afe54af4360114cd"
I20260812 11:14:56.452603 31493 raft_consensus.cc:3037] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.452553 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f206a5477b694c20acba87ef049b4750" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a"
I20260812 11:14:56.452798 31284 raft_consensus.cc:3037] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.453967 31493 raft_consensus.cc:2454] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 1.
I20260812 11:14:56.454309 31284 raft_consensus.cc:2454] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 1.
I20260812 11:14:56.454602 30976 leader_election.cc:302] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07037b345dfc454eb0514632d807681d, 2201be271dff4291afe54af4360114cd; no voters:
I20260812 11:14:56.455018 31601 raft_consensus.cc:2781] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:56.455271 31601 raft_consensus.cc:686] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [term 1 LEADER]: Becoming Leader. State: Replica: 07037b345dfc454eb0514632d807681d, State: Running, Role: LEADER
I20260812 11:14:56.455834 31601 consensus_queue.cc:237] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.459134 31628 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: TxnStatusTablet state changed. Reason: New leader 07037b345dfc454eb0514632d807681d. Latest consensus state: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.459337 31628 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:14:56.459698 31630 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:14:56.459962 31630 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:14:56.461233 30835 catalog_manager.cc:5495] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d reported cstate change: term changed from 0 to 1, leader changed from <none> to 07037b345dfc454eb0514632d807681d (127.27.109.1). New cstate: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } health_report { overall_health: UNKNOWN } } }
I20260812 11:14:56.480659 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.475219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.27.109.1:32895)
0812 11:14:56.475380 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.475389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.475535 (+ 146us) server_negotiation.cc:407] Connection header received
0812 11:14:56.475882 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.475885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.476055 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.476214 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:56.476598 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.477150 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.478661 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.478986 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.479080 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.479584 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:56.480142 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:56.480243 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.480414 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.480523 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:14:56.480737 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.475131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:32895 (local address 127.0.0.1:40286)
0812 11:14:56.475380 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.475405 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.475633 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.476250 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.476254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.476264 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:14:56.476448 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.476459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.477291 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.477295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.478497 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.478510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.479148 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.479153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.479270 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.479358 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:14:56.480269 (+ 911us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:14:56.480289 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.480385 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.480536 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 11:14:56.487162 31284 raft_consensus.cc:1261] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Refusing update from remote peer 07037b345dfc454eb0514632d807681d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:56.487213 31493 raft_consensus.cc:1261] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Refusing update from remote peer 07037b345dfc454eb0514632d807681d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:56.487926 31601 consensus_queue.cc:1035] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:56.488267 31628 consensus_queue.cc:1035] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:56.492291 31603 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: TxnStatusTablet state changed. Reason: New leader 07037b345dfc454eb0514632d807681d. Latest consensus state: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.492440 31603 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:56.494722 31602 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: TxnStatusTablet state changed. Reason: New leader 07037b345dfc454eb0514632d807681d. Latest consensus state: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.494951 31602 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:56.496893 31628 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.497030 31628 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:14:56.497298 31636 mvcc.cc:205] Tried to move back new op lower bound from 7317640382401880064 to 7317640382286671872. Current Snapshot: MvccSnapshot[applied={T|T < 7317640382401880064}]
I20260812 11:14:56.498143 31642 mvcc.cc:205] Tried to move back new op lower bound from 7317640382401880064 to 7317640382286671872. Current Snapshot: MvccSnapshot[applied={T|T < 7317640382401880064}]
I20260812 11:14:56.498342 31602 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.498466 31602 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P d0495a0f57ca472ca094269eb943d46a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:56.499643 31601 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.499874 31601 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P 07037b345dfc454eb0514632d807681d: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:14:56.501993 31603 tablet_replica.cc:439] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } } }
I20260812 11:14:56.502162 31603 tablet_replica.cc:442] T f206a5477b694c20acba87ef049b4750 P 2201be271dff4291afe54af4360114cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:14:56.502553 31639 mvcc.cc:205] Tried to move back new op lower bound from 7317640382401880064 to 7317640382286671872. Current Snapshot: MvccSnapshot[applied={T|T < 7317640382401880064}]
I20260812 11:14:56.512619 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.508784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.27.109.3:42839)
0812 11:14:56.508969 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.508977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.508990 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:14:56.509067 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.509069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.509126 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.509247 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:56.509564 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.510101 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.511059 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.511283 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.511400 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.511703 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:56.512204 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:56.512332 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.512405 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.512465 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:14:56.526104 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.522709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:34049 (local address 127.27.109.1:32895)
0812 11:14:56.522942 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.522947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.522962 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:14:56.523011 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.523014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.523057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.523156 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:56.523432 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.523968 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.524818 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.525059 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.525172 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.525564 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:56.525647 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:56.525738 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.525883 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.525948 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 11:14:56.526104 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.522398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:32895 (local address 127.27.109.3:34049)
0812 11:14:56.522582 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.522600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.522699 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.523168 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.523172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.523179 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:14:56.523315 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.523321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.524085 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.524090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.524685 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.524693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.525191 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.525195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.525289 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.525345 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:14:56.525745 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:14:56.525760 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.525862 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.525973 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 11:14:56.535959 31354 tablet.cc:2369] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:14:56.537235 31284 raft_consensus.cc:1261] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Refusing update from remote peer 2201be271dff4291afe54af4360114cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:56.537490 31074 raft_consensus.cc:1261] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Refusing update from remote peer 2201be271dff4291afe54af4360114cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:56.537860 31604 consensus_queue.cc:1035] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:56.538168 31603 consensus_queue.cc:1035] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:56.573297 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.567728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42839 (local address 127.27.109.1:50623)
0812 11:14:56.568282 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.568305 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:56.568576 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:56.569026 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:56.569030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:56.569040 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:14:56.569249 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.569256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.570165 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.570170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.571258 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:56.571274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.572056 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.572061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:56.572159 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.572233 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:14:56.572834 (+ 601us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:14:56.572845 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 11:14:56.572921 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 11:14:56.573053 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
I20260812 11:14:56.573443 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.568405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50623 (local address 127.27.109.3:42839)
0812 11:14:56.568575 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:56.568582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:56.568601 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:14:56.568793 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:56.568798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:56.568892 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:56.569007 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:56.569390 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.570019 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.571439 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:56.571850 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:56.572008 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:56.572601 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:56.572729 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:56.572815 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:56.573194 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 11:14:56.573281 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:14:56.614207 30835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:56.614480 30835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:56.619069 30835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Sending DeleteTablet for 3 replicas of tablet 773696bb5665486da5e4041022ea99f0
I20260812 11:14:56.620723 31246 tablet_service.cc:1514] Processing DeleteTablet for tablet 773696bb5665486da5e4041022ea99f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:56 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42854
I20260812 11:14:56.620913 31461 tablet_service.cc:1514] Processing DeleteTablet for tablet 773696bb5665486da5e4041022ea99f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:56 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49820
I20260812 11:14:56.621423 31664 tablet_replica.cc:330] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: stopping tablet replica
I20260812 11:14:56.621833 31664 raft_consensus.cc:2229] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:56.622135 31664 raft_consensus.cc:2258] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:56.622803 31049 tablet_service.cc:1514] Processing DeleteTablet for tablet 773696bb5665486da5e4041022ea99f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:56 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40278
I20260812 11:14:56.622951 30835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:56.623528 30835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:56.623833 31665 tablet_replica.cc:330] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: stopping tablet replica
I20260812 11:14:56.626108 31664 ts_tablet_manager.cc:1905] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:56.631405 31049 tablet_service.cc:1467] Processing CreateTablet for tablet 7418beef1e584b5eaf65f846978ee419 (DEFAULT_TABLE table=test-table [id=3e46b6048f0a4616a4aa4b00a8a8cb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:56.631881 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7418beef1e584b5eaf65f846978ee419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.634718 31664 ts_tablet_manager.cc:1918] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:56.634894 31664 log.cc:1196] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/773696bb5665486da5e4041022ea99f0
I20260812 11:14:56.635325 31664 ts_tablet_manager.cc:1939] T 773696bb5665486da5e4041022ea99f0 P d0495a0f57ca472ca094269eb943d46a: Deleting consensus metadata
I20260812 11:14:56.636103 31246 tablet_service.cc:1467] Processing CreateTablet for tablet 7418beef1e584b5eaf65f846978ee419 (DEFAULT_TABLE table=test-table [id=3e46b6048f0a4616a4aa4b00a8a8cb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:56.636538 31593 tablet_bootstrap.cc:492] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: Bootstrap starting.
I20260812 11:14:56.637197 31246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7418beef1e584b5eaf65f846978ee419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.637787 31593 tablet_bootstrap.cc:654] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.635282 31461 tablet_service.cc:1467] Processing CreateTablet for tablet 7418beef1e584b5eaf65f846978ee419 (DEFAULT_TABLE table=test-table [id=3e46b6048f0a4616a4aa4b00a8a8cb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:56.640053 31461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7418beef1e584b5eaf65f846978ee419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:56.641260 31595 tablet_bootstrap.cc:492] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: Bootstrap starting.
I20260812 11:14:56.642455 31595 tablet_bootstrap.cc:654] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.643801 31592 tablet_bootstrap.cc:492] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: Bootstrap starting.
I20260812 11:14:56.645170 31592 tablet_bootstrap.cc:654] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:56.645917 30822 catalog_manager.cc:4841] TS d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915): tablet 773696bb5665486da5e4041022ea99f0 (table test-table [id=94f93cd7ef1d48349bde6a13c0310ed8]) successfully deleted
I20260812 11:14:56.624351 31665 raft_consensus.cc:2229] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:14:56.646996 31595 tablet_bootstrap.cc:492] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: No bootstrap required, opened a new log
I20260812 11:14:56.647104 31595 ts_tablet_manager.cc:1397] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260812 11:14:56.647569 31595 raft_consensus.cc:348] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.647615 31670 tablet_replica.cc:330] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: stopping tablet replica
I20260812 11:14:56.647702 31595 raft_consensus.cc:374] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.647746 31595 raft_consensus.cc:729] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0495a0f57ca472ca094269eb943d46a, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.647884 31595 consensus_queue.cc:260] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.648144 31595 ts_tablet_manager.cc:1428] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 11:14:56.648444 31665 raft_consensus.cc:2258] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:56.649014 31670 raft_consensus.cc:2229] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:56.649560 31670 raft_consensus.cc:2258] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:56.651729 31670 ts_tablet_manager.cc:1905] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:56.651729 31665 ts_tablet_manager.cc:1905] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:56.656206 31592 tablet_bootstrap.cc:492] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: No bootstrap required, opened a new log
I20260812 11:14:56.656373 31592 ts_tablet_manager.cc:1397] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260812 11:14:56.657068 31592 raft_consensus.cc:348] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.657231 31592 raft_consensus.cc:374] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.657265 31592 raft_consensus.cc:729] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2201be271dff4291afe54af4360114cd, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.657436 31592 consensus_queue.cc:260] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.657754 31592 ts_tablet_manager.cc:1428] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:14:56.660252 31593 tablet_bootstrap.cc:492] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: No bootstrap required, opened a new log
I20260812 11:14:56.660396 31593 ts_tablet_manager.cc:1397] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20260812 11:14:56.660984 31593 raft_consensus.cc:348] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.661144 31593 raft_consensus.cc:374] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:56.661177 31593 raft_consensus.cc:729] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07037b345dfc454eb0514632d807681d, State: Initialized, Role: FOLLOWER
I20260812 11:14:56.661363 31593 consensus_queue.cc:260] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.661696 31593 ts_tablet_manager.cc:1428] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:14:56.662034 31670 ts_tablet_manager.cc:1918] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:56.662187 31670 log.cc:1196] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/773696bb5665486da5e4041022ea99f0
I20260812 11:14:56.662721 31670 ts_tablet_manager.cc:1939] T 773696bb5665486da5e4041022ea99f0 P 07037b345dfc454eb0514632d807681d: Deleting consensus metadata
I20260812 11:14:56.665253 30824 catalog_manager.cc:4841] TS 07037b345dfc454eb0514632d807681d (127.27.109.1:32895): tablet 773696bb5665486da5e4041022ea99f0 (table test-table [id=94f93cd7ef1d48349bde6a13c0310ed8]) successfully deleted
I20260812 11:14:56.667276 31665 ts_tablet_manager.cc:1918] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:56.667517 31665 log.cc:1196] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/773696bb5665486da5e4041022ea99f0
I20260812 11:14:56.668097 31665 ts_tablet_manager.cc:1939] T 773696bb5665486da5e4041022ea99f0 P 2201be271dff4291afe54af4360114cd: Deleting consensus metadata
I20260812 11:14:56.669947 30822 catalog_manager.cc:4841] TS 2201be271dff4291afe54af4360114cd (127.27.109.3:42839): tablet 773696bb5665486da5e4041022ea99f0 (table test-table [id=94f93cd7ef1d48349bde6a13c0310ed8]) successfully deleted
I20260812 11:14:56.673386 31656 raft_consensus.cc:482] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:56.673619 31656 raft_consensus.cc:504] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.674206 31656 leader_election.cc:288] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07037b345dfc454eb0514632d807681d (127.27.109.1:32895), d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915)
I20260812 11:14:56.675117 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7418beef1e584b5eaf65f846978ee419" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a" is_pre_election: true
I20260812 11:14:56.675287 31284 raft_consensus.cc:2454] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 0.
I20260812 11:14:56.675531 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7418beef1e584b5eaf65f846978ee419" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07037b345dfc454eb0514632d807681d" is_pre_election: true
I20260812 11:14:56.675693 31074 raft_consensus.cc:2454] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 0.
I20260812 11:14:56.676113 31400 leader_election.cc:302] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07037b345dfc454eb0514632d807681d, 2201be271dff4291afe54af4360114cd; no voters:
I20260812 11:14:56.676455 31656 raft_consensus.cc:2781] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:56.676519 31656 raft_consensus.cc:482] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:56.676570 31656 raft_consensus.cc:3037] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.678223 31656 raft_consensus.cc:504] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.678725 31656 leader_election.cc:288] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 election: Requested vote from peers 07037b345dfc454eb0514632d807681d (127.27.109.1:32895), d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915)
I20260812 11:14:56.679214 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7418beef1e584b5eaf65f846978ee419" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a"
I20260812 11:14:56.679402 31284 raft_consensus.cc:3037] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.679342 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7418beef1e584b5eaf65f846978ee419" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07037b345dfc454eb0514632d807681d"
I20260812 11:14:56.679623 31074 raft_consensus.cc:3037] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:56.680729 31284 raft_consensus.cc:2454] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 1.
I20260812 11:14:56.681200 31399 leader_election.cc:302] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2201be271dff4291afe54af4360114cd, d0495a0f57ca472ca094269eb943d46a; no voters:
I20260812 11:14:56.681430 31656 raft_consensus.cc:2781] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:56.681565 31656 raft_consensus.cc:686] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 1 LEADER]: Becoming Leader. State: Replica: 2201be271dff4291afe54af4360114cd, State: Running, Role: LEADER
I20260812 11:14:56.681667 31074 raft_consensus.cc:2454] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 1.
I20260812 11:14:56.681734 31656 consensus_queue.cc:237] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:56.683825 30835 catalog_manager.cc:5495] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2201be271dff4291afe54af4360114cd (127.27.109.3). New cstate: current_term: 1 leader_uuid: "2201be271dff4291afe54af4360114cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } health_report { overall_health: HEALTHY } } }
I20260812 11:14:56.711418 31284 raft_consensus.cc:1261] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Refusing update from remote peer 2201be271dff4291afe54af4360114cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:56.711875 31074 raft_consensus.cc:1261] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Refusing update from remote peer 2201be271dff4291afe54af4360114cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:56.712218 31604 consensus_queue.cc:1035] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:56.712404 31656 consensus_queue.cc:1035] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:14:56.733781 31639 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 11:14:58.326426 30835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:58.326706 30835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:58.331001 30835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Sending DeleteTablet for 3 replicas of tablet 7418beef1e584b5eaf65f846978ee419
I20260812 11:14:58.332194 31049 tablet_service.cc:1514] Processing DeleteTablet for tablet 7418beef1e584b5eaf65f846978ee419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40278
I20260812 11:14:58.332342 31246 tablet_service.cc:1514] Processing DeleteTablet for tablet 7418beef1e584b5eaf65f846978ee419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42854
I20260812 11:14:58.332322 31461 tablet_service.cc:1514] Processing DeleteTablet for tablet 7418beef1e584b5eaf65f846978ee419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49820
I20260812 11:14:58.332789 31708 tablet_replica.cc:330] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: stopping tablet replica
I20260812 11:14:58.332789 31709 tablet_replica.cc:330] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: stopping tablet replica
I20260812 11:14:58.332957 31709 raft_consensus.cc:2229] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:58.332957 31708 raft_consensus.cc:2229] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:58.333122 31708 raft_consensus.cc:2258] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:58.333122 31709 raft_consensus.cc:2258] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:58.333483 31711 tablet_replica.cc:330] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: stopping tablet replica
I20260812 11:14:58.333647 31711 raft_consensus.cc:2229] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:14:58.333956 31711 raft_consensus.cc:2258] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:58.334339 31708 ts_tablet_manager.cc:1905] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:58.334478 31709 ts_tablet_manager.cc:1905] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:58.334772 31711 ts_tablet_manager.cc:1905] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:58.338271 31709 ts_tablet_manager.cc:1918] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:14:58.338271 31711 ts_tablet_manager.cc:1918] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:14:58.338294 31708 ts_tablet_manager.cc:1918] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:14:58.338377 31711 log.cc:1196] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/7418beef1e584b5eaf65f846978ee419
I20260812 11:14:58.338377 31709 log.cc:1196] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/7418beef1e584b5eaf65f846978ee419
I20260812 11:14:58.338403 31708 log.cc:1196] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/7418beef1e584b5eaf65f846978ee419
I20260812 11:14:58.338760 31709 ts_tablet_manager.cc:1939] T 7418beef1e584b5eaf65f846978ee419 P d0495a0f57ca472ca094269eb943d46a: Deleting consensus metadata
I20260812 11:14:58.338760 31711 ts_tablet_manager.cc:1939] T 7418beef1e584b5eaf65f846978ee419 P 2201be271dff4291afe54af4360114cd: Deleting consensus metadata
I20260812 11:14:58.338814 31708 ts_tablet_manager.cc:1939] T 7418beef1e584b5eaf65f846978ee419 P 07037b345dfc454eb0514632d807681d: Deleting consensus metadata
I20260812 11:14:58.340013 30822 catalog_manager.cc:4841] TS d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915): tablet 7418beef1e584b5eaf65f846978ee419 (table test-table [id=3e46b6048f0a4616a4aa4b00a8a8cb82]) successfully deleted
I20260812 11:14:58.340045 30824 catalog_manager.cc:4841] TS 07037b345dfc454eb0514632d807681d (127.27.109.1:32895): tablet 7418beef1e584b5eaf65f846978ee419 (table test-table [id=3e46b6048f0a4616a4aa4b00a8a8cb82]) successfully deleted
I20260812 11:14:58.340335 30822 catalog_manager.cc:4841] TS 2201be271dff4291afe54af4360114cd (127.27.109.3:42839): tablet 7418beef1e584b5eaf65f846978ee419 (table test-table [id=3e46b6048f0a4616a4aa4b00a8a8cb82]) successfully deleted
Aug 12 11:14:58 dist-test-slave-7z32 krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:14:58 dist-test-slave-7z32 krb5kdc[30779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:14:58.380623 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.365094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.27.109.62:46505)
0812 11:14:58.365544 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:58.365551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:58.365566 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:14:58.365613 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:58.365616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:58.365659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:58.365768 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:58.366574 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.367198 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.368047 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.368270 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.368380 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:58.376954 (+ 8574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:58.376959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:58.376961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:58.377000 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:58.378643 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:58.379201 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:58.379205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:58.379206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:58.379294 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:58.379647 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:58.379651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:58.379652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:58.379873 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:58.380039 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:58.380283 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 11:14:58.380417 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":176,"threads_started":1}
I20260812 11:14:58.384976 30835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:58.385807 30835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:58.393683 31461 tablet_service.cc:1467] Processing CreateTablet for tablet e76ba5e78a11429e88b943a31e02011d (DEFAULT_TABLE table=test-table [id=268ac899c14b497ea84d70bca6b129a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:58.393765 31049 tablet_service.cc:1467] Processing CreateTablet for tablet e76ba5e78a11429e88b943a31e02011d (DEFAULT_TABLE table=test-table [id=268ac899c14b497ea84d70bca6b129a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:58.393728 31246 tablet_service.cc:1467] Processing CreateTablet for tablet e76ba5e78a11429e88b943a31e02011d (DEFAULT_TABLE table=test-table [id=268ac899c14b497ea84d70bca6b129a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:58.394145 31461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e76ba5e78a11429e88b943a31e02011d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:58.394192 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e76ba5e78a11429e88b943a31e02011d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:58.394192 31246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e76ba5e78a11429e88b943a31e02011d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:58.396816 31722 tablet_bootstrap.cc:492] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: Bootstrap starting.
I20260812 11:14:58.398026 31722 tablet_bootstrap.cc:654] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:58.398065 31723 tablet_bootstrap.cc:492] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: Bootstrap starting.
I20260812 11:14:58.398325 31726 tablet_bootstrap.cc:492] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: Bootstrap starting.
I20260812 11:14:58.399632 31726 tablet_bootstrap.cc:654] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:58.399817 31722 tablet_bootstrap.cc:492] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: No bootstrap required, opened a new log
I20260812 11:14:58.399948 31722 ts_tablet_manager.cc:1397] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:58.400559 31722 raft_consensus.cc:348] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.400717 31722 raft_consensus.cc:374] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:58.400756 31722 raft_consensus.cc:729] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2201be271dff4291afe54af4360114cd, State: Initialized, Role: FOLLOWER
I20260812 11:14:58.400992 31723 tablet_bootstrap.cc:654] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:58.400986 31722 consensus_queue.cc:260] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.401405 31722 ts_tablet_manager.cc:1428] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:14:58.401835 31726 tablet_bootstrap.cc:492] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: No bootstrap required, opened a new log
I20260812 11:14:58.402048 31726 ts_tablet_manager.cc:1397] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 11:14:58.402665 31726 raft_consensus.cc:348] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.402833 31726 raft_consensus.cc:374] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:58.402917 31726 raft_consensus.cc:729] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07037b345dfc454eb0514632d807681d, State: Initialized, Role: FOLLOWER
I20260812 11:14:58.403124 31726 consensus_queue.cc:260] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.403285 31681 raft_consensus.cc:482] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:58.403317 31723 tablet_bootstrap.cc:492] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: No bootstrap required, opened a new log
I20260812 11:14:58.403360 31681 raft_consensus.cc:504] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.403429 31723 ts_tablet_manager.cc:1397] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:14:58.403966 31681 leader_election.cc:288] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915), 2201be271dff4291afe54af4360114cd (127.27.109.3:42839)
I20260812 11:14:58.403980 31723 raft_consensus.cc:348] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.404114 31723 raft_consensus.cc:374] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:58.404147 31723 raft_consensus.cc:729] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0495a0f57ca472ca094269eb943d46a, State: Initialized, Role: FOLLOWER
I20260812 11:14:58.404287 31723 consensus_queue.cc:260] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.404785 31723 ts_tablet_manager.cc:1428] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:14:58.404765 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ba5e78a11429e88b943a31e02011d" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a" is_pre_election: true
I20260812 11:14:58.404995 31284 raft_consensus.cc:2454] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 0.
I20260812 11:14:58.405282 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ba5e78a11429e88b943a31e02011d" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2201be271dff4291afe54af4360114cd" is_pre_election: true
I20260812 11:14:58.405442 31493 raft_consensus.cc:2454] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 0.
I20260812 11:14:58.405536 30976 leader_election.cc:302] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07037b345dfc454eb0514632d807681d, d0495a0f57ca472ca094269eb943d46a; no voters:
I20260812 11:14:58.405779 31681 raft_consensus.cc:2781] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:58.405853 31681 raft_consensus.cc:482] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:58.405897 31681 raft_consensus.cc:3037] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:58.406527 31726 ts_tablet_manager.cc:1428] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260812 11:14:58.407581 31681 raft_consensus.cc:504] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.408071 31681 leader_election.cc:288] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 election: Requested vote from peers d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915), 2201be271dff4291afe54af4360114cd (127.27.109.3:42839)
I20260812 11:14:58.408566 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ba5e78a11429e88b943a31e02011d" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2201be271dff4291afe54af4360114cd"
I20260812 11:14:58.408680 31493 raft_consensus.cc:3037] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:58.409683 31493 raft_consensus.cc:2454] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 1.
I20260812 11:14:58.410005 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ba5e78a11429e88b943a31e02011d" candidate_uuid: "07037b345dfc454eb0514632d807681d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a"
I20260812 11:14:58.410140 31284 raft_consensus.cc:3037] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:58.410073 30976 leader_election.cc:302] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07037b345dfc454eb0514632d807681d, 2201be271dff4291afe54af4360114cd; no voters:
I20260812 11:14:58.410480 31681 raft_consensus.cc:2781] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:58.410663 31681 raft_consensus.cc:686] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 1 LEADER]: Becoming Leader. State: Replica: 07037b345dfc454eb0514632d807681d, State: Running, Role: LEADER
I20260812 11:14:58.410953 31681 consensus_queue.cc:237] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.411459 31284 raft_consensus.cc:2454] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07037b345dfc454eb0514632d807681d in term 1.
I20260812 11:14:58.415586 30835 catalog_manager.cc:5495] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d reported cstate change: term changed from 0 to 1, leader changed from <none> to 07037b345dfc454eb0514632d807681d (127.27.109.1). New cstate: current_term: 1 leader_uuid: "07037b345dfc454eb0514632d807681d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } health_report { overall_health: UNKNOWN } } }
I20260812 11:14:58.442066 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.438390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.27.109.1:32895)
0812 11:14:58.438809 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:58.438869 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:58.438886 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:14:58.438942 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:58.438945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:58.438994 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:58.439090 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:58.439391 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.439991 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.440909 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.441152 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.441265 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:58.441557 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:58.441644 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:58.441726 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:58.441824 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 11:14:58.441885 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":194,"threads_started":1}
I20260812 11:14:58.453860 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.450331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:32895 (local address 127.27.109.1:56033)
0812 11:14:58.450719 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:58.450734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:58.450936 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:58.451187 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:58.451190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:58.451197 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:14:58.451319 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:58.451332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.452066 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.452072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:58.452682 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:58.452690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.453137 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.453141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:58.453232 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:58.453263 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:14:58.453552 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:14:58.453560 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:14:58.453622 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 11:14:58.453713 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":1,"thread_start_us":186,"threads_started":1}
I20260812 11:14:58.453859 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.450432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:56033 (local address 127.27.109.1:32895)
0812 11:14:58.450601 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:58.450606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:58.450860 (+ 254us) server_negotiation.cc:407] Connection header received
0812 11:14:58.451045 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:58.451048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:58.451100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:58.451178 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:58.451441 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.451960 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.452804 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.453027 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.453135 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:58.453378 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:58.453462 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:58.453537 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:58.453630 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 11:14:58.453688 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":57}
I20260812 11:14:58.462088 31493 raft_consensus.cc:1261] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Refusing update from remote peer 07037b345dfc454eb0514632d807681d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:58.462157 31284 raft_consensus.cc:1261] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Refusing update from remote peer 07037b345dfc454eb0514632d807681d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:58.462750 31678 consensus_queue.cc:1035] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:58.463172 31747 consensus_queue.cc:1035] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:58.513535 30843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:58.513768 30843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:58.517038 30843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Sending DeleteTablet for 3 replicas of tablet e76ba5e78a11429e88b943a31e02011d
I20260812 11:14:58.518218 31246 tablet_service.cc:1514] Processing DeleteTablet for tablet e76ba5e78a11429e88b943a31e02011d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42854
I20260812 11:14:58.518412 31461 tablet_service.cc:1514] Processing DeleteTablet for tablet e76ba5e78a11429e88b943a31e02011d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49820
I20260812 11:14:58.518551 31709 tablet_replica.cc:330] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: stopping tablet replica
I20260812 11:14:58.518682 31711 tablet_replica.cc:330] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: stopping tablet replica
I20260812 11:14:58.518728 31709 raft_consensus.cc:2229] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:58.518802 31711 raft_consensus.cc:2229] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:58.518960 31709 raft_consensus.cc:2258] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:58.518968 31711 raft_consensus.cc:2258] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:58.519080 30843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:58.519622 30843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:58.520007 31709 ts_tablet_manager.cc:1905] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:58.520299 31711 ts_tablet_manager.cc:1905] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:58.520495 31049 tablet_service.cc:1514] Processing DeleteTablet for tablet e76ba5e78a11429e88b943a31e02011d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40278
I20260812 11:14:58.520717 31708 tablet_replica.cc:330] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: stopping tablet replica
I20260812 11:14:58.520859 31708 raft_consensus.cc:2229] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:14:58.521250 31708 raft_consensus.cc:2258] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:58.522189 31708 ts_tablet_manager.cc:1905] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:58.523577 31711 ts_tablet_manager.cc:1918] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:58.523567 31709 ts_tablet_manager.cc:1918] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:58.523670 31709 log.cc:1196] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/e76ba5e78a11429e88b943a31e02011d
I20260812 11:14:58.523670 31711 log.cc:1196] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/e76ba5e78a11429e88b943a31e02011d
I20260812 11:14:58.523985 31711 ts_tablet_manager.cc:1939] T e76ba5e78a11429e88b943a31e02011d P 2201be271dff4291afe54af4360114cd: Deleting consensus metadata
I20260812 11:14:58.523985 31709 ts_tablet_manager.cc:1939] T e76ba5e78a11429e88b943a31e02011d P d0495a0f57ca472ca094269eb943d46a: Deleting consensus metadata
I20260812 11:14:58.525208 31708 ts_tablet_manager.cc:1918] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:14:58.525327 31708 log.cc:1196] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/e76ba5e78a11429e88b943a31e02011d
I20260812 11:14:58.525533 30822 catalog_manager.cc:4841] TS 2201be271dff4291afe54af4360114cd (127.27.109.3:42839): tablet e76ba5e78a11429e88b943a31e02011d (table test-table [id=268ac899c14b497ea84d70bca6b129a2]) successfully deleted
I20260812 11:14:58.525629 31708 ts_tablet_manager.cc:1939] T e76ba5e78a11429e88b943a31e02011d P 07037b345dfc454eb0514632d807681d: Deleting consensus metadata
I20260812 11:14:58.526952 30822 catalog_manager.cc:4841] TS d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915): tablet e76ba5e78a11429e88b943a31e02011d (table test-table [id=268ac899c14b497ea84d70bca6b129a2]) successfully deleted
I20260812 11:14:58.527737 30824 catalog_manager.cc:4841] TS 07037b345dfc454eb0514632d807681d (127.27.109.1:32895): tablet e76ba5e78a11429e88b943a31e02011d (table test-table [id=268ac899c14b497ea84d70bca6b129a2]) successfully deleted
I20260812 11:14:58.530368 31049 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3d4634db8c4ba79ae8294050d80a3e (DEFAULT_TABLE table=test-table [id=9b2743925d4f4b21adeb48c2092a52c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:58.530679 31246 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3d4634db8c4ba79ae8294050d80a3e (DEFAULT_TABLE table=test-table [id=9b2743925d4f4b21adeb48c2092a52c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:58.531132 31246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3d4634db8c4ba79ae8294050d80a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:58.530740 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3d4634db8c4ba79ae8294050d80a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:58.534112 31726 tablet_bootstrap.cc:492] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d: Bootstrap starting.
I20260812 11:14:58.534651 31723 tablet_bootstrap.cc:492] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: Bootstrap starting.
I20260812 11:14:58.535531 31726 tablet_bootstrap.cc:654] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:58.535449 31461 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3d4634db8c4ba79ae8294050d80a3e (DEFAULT_TABLE table=test-table [id=9b2743925d4f4b21adeb48c2092a52c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:58.535821 31461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3d4634db8c4ba79ae8294050d80a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:58.535871 31723 tablet_bootstrap.cc:654] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:58.537194 31723 tablet_bootstrap.cc:492] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: No bootstrap required, opened a new log
I20260812 11:14:58.537407 31723 ts_tablet_manager.cc:1397] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:58.537897 31723 raft_consensus.cc:348] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.538043 31723 raft_consensus.cc:374] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:58.538075 31723 raft_consensus.cc:729] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0495a0f57ca472ca094269eb943d46a, State: Initialized, Role: FOLLOWER
I20260812 11:14:58.538214 31723 consensus_queue.cc:260] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.538528 31723 ts_tablet_manager.cc:1428] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:14:58.538935 31722 tablet_bootstrap.cc:492] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: Bootstrap starting.
I20260812 11:14:58.540025 31722 tablet_bootstrap.cc:654] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:58.540248 31726 tablet_bootstrap.cc:492] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d: No bootstrap required, opened a new log
I20260812 11:14:58.540364 31726 ts_tablet_manager.cc:1397] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260812 11:14:58.540815 31726 raft_consensus.cc:348] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.540947 31726 raft_consensus.cc:374] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:58.540989 31726 raft_consensus.cc:729] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07037b345dfc454eb0514632d807681d, State: Initialized, Role: FOLLOWER
I20260812 11:14:58.541154 31726 consensus_queue.cc:260] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.541481 31726 ts_tablet_manager.cc:1428] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:14:58.541846 31722 tablet_bootstrap.cc:492] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: No bootstrap required, opened a new log
I20260812 11:14:58.541978 31722 ts_tablet_manager.cc:1397] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:58.542426 31722 raft_consensus.cc:348] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.542577 31722 raft_consensus.cc:374] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:58.542619 31722 raft_consensus.cc:729] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2201be271dff4291afe54af4360114cd, State: Initialized, Role: FOLLOWER
I20260812 11:14:58.542793 31722 consensus_queue.cc:260] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.543326 31722 ts_tablet_manager.cc:1428] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:14:58.856274 31656 raft_consensus.cc:482] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:58.856552 31656 raft_consensus.cc:504] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.857288 31656 leader_election.cc:288] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07037b345dfc454eb0514632d807681d (127.27.109.1:32895), d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915)
I20260812 11:14:58.857947 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3d4634db8c4ba79ae8294050d80a3e" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07037b345dfc454eb0514632d807681d" is_pre_election: true
I20260812 11:14:58.858031 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3d4634db8c4ba79ae8294050d80a3e" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a" is_pre_election: true
I20260812 11:14:58.858168 31074 raft_consensus.cc:2454] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 0.
I20260812 11:14:58.858297 31284 raft_consensus.cc:2454] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 0.
I20260812 11:14:58.858623 31400 leader_election.cc:302] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07037b345dfc454eb0514632d807681d, 2201be271dff4291afe54af4360114cd; no voters:
I20260812 11:14:58.858968 31656 raft_consensus.cc:2781] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:58.859061 31656 raft_consensus.cc:482] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:58.859102 31656 raft_consensus.cc:3037] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:58.860344 31656 raft_consensus.cc:504] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.860818 31656 leader_election.cc:288] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 election: Requested vote from peers 07037b345dfc454eb0514632d807681d (127.27.109.1:32895), d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915)
I20260812 11:14:58.861341 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3d4634db8c4ba79ae8294050d80a3e" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07037b345dfc454eb0514632d807681d"
I20260812 11:14:58.861436 31284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3d4634db8c4ba79ae8294050d80a3e" candidate_uuid: "2201be271dff4291afe54af4360114cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0495a0f57ca472ca094269eb943d46a"
I20260812 11:14:58.861582 31284 raft_consensus.cc:3037] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:58.861675 31074 raft_consensus.cc:3037] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:58.862802 31284 raft_consensus.cc:2454] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 1.
I20260812 11:14:58.863265 31399 leader_election.cc:302] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2201be271dff4291afe54af4360114cd, d0495a0f57ca472ca094269eb943d46a; no voters:
I20260812 11:14:58.863544 31074 raft_consensus.cc:2454] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2201be271dff4291afe54af4360114cd in term 1.
I20260812 11:14:58.863582 31656 raft_consensus.cc:2781] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:58.863816 31656 raft_consensus.cc:686] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 1 LEADER]: Becoming Leader. State: Replica: 2201be271dff4291afe54af4360114cd, State: Running, Role: LEADER
I20260812 11:14:58.864022 31656 consensus_queue.cc:237] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } }
I20260812 11:14:58.866070 30839 catalog_manager.cc:5495] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2201be271dff4291afe54af4360114cd (127.27.109.3). New cstate: current_term: 1 leader_uuid: "2201be271dff4291afe54af4360114cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2201be271dff4291afe54af4360114cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42839 } health_report { overall_health: HEALTHY } } }
I20260812 11:14:58.932794 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.928511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.27.109.3:42839)
0812 11:14:58.928864 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:58.928893 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:58.928909 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:14:58.928973 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:58.928976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:58.929034 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:58.929146 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:58.929468 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.930084 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.931187 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:58.931541 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:58.931654 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:58.932048 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:58.932216 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:58.932408 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:58.932503 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 11:14:58.932584 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260812 11:14:58.942368 31074 raft_consensus.cc:1261] T 1e3d4634db8c4ba79ae8294050d80a3e P 07037b345dfc454eb0514632d807681d [term 1 FOLLOWER]: Refusing update from remote peer 2201be271dff4291afe54af4360114cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:58.942453 31284 raft_consensus.cc:1261] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Refusing update from remote peer 2201be271dff4291afe54af4360114cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:58.943027 31760 consensus_queue.cc:1035] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07037b345dfc454eb0514632d807681d" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:58.943261 31656 consensus_queue.cc:1035] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0495a0f57ca472ca094269eb943d46a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:14:58.962167 31684 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 11:15:00.544382 30843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:00.544768 30843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:00.548979 30843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 837038e80c7545c1ba933a5308c9be64: Sending DeleteTablet for 3 replicas of tablet 1e3d4634db8c4ba79ae8294050d80a3e
I20260812 11:15:00.549929 31049 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3d4634db8c4ba79ae8294050d80a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:00 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40278
I20260812 11:15:00.550036 31246 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3d4634db8c4ba79ae8294050d80a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:00 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42854
I20260812 11:15:00.550160 31461 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3d4634db8c4ba79ae8294050d80a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:00 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49820
I20260812 11:15:00.550344 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30955
I20260812 11:15:00.550601 31807 tablet_replica.cc:330] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: stopping tablet replica
I20260812 11:15:00.550845 31807 raft_consensus.cc:2229] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:00.551141 31807 raft_consensus.cc:2258] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:00.551994 31808 tablet_replica.cc:330] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: stopping tablet replica
I20260812 11:15:00.552205 31808 raft_consensus.cc:2229] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:00.552505 31808 raft_consensus.cc:2258] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:00.559234 31808 ts_tablet_manager.cc:1905] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:00.563969 31808 ts_tablet_manager.cc:1918] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:15:00.564107 31808 log.cc:1196] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/1e3d4634db8c4ba79ae8294050d80a3e
I20260812 11:15:00.564436 31808 ts_tablet_manager.cc:1939] T 1e3d4634db8c4ba79ae8294050d80a3e P 2201be271dff4291afe54af4360114cd: Deleting consensus metadata
I20260812 11:15:00.565660 30822 catalog_manager.cc:4841] TS 2201be271dff4291afe54af4360114cd (127.27.109.3:42839): tablet 1e3d4634db8c4ba79ae8294050d80a3e (table test-table [id=9b2743925d4f4b21adeb48c2092a52c1]) successfully deleted
I20260812 11:15:00.569588 31807 ts_tablet_manager.cc:1905] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:00.572707 31807 ts_tablet_manager.cc:1918] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:15:00.572824 31807 log.cc:1196] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/1e3d4634db8c4ba79ae8294050d80a3e
I20260812 11:15:00.573132 31807 ts_tablet_manager.cc:1939] T 1e3d4634db8c4ba79ae8294050d80a3e P d0495a0f57ca472ca094269eb943d46a: Deleting consensus metadata
I20260812 11:15:00.574391 30822 catalog_manager.cc:4841] TS d0495a0f57ca472ca094269eb943d46a (127.27.109.2:44915): tablet 1e3d4634db8c4ba79ae8294050d80a3e (table test-table [id=9b2743925d4f4b21adeb48c2092a52c1]) successfully deleted
W20260812 11:15:00.577386 30824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:32895 (error 108)
I20260812 11:15:00.577826 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31153
I20260812 11:15:00.585316 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:00.582249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:32895 (local address 127.0.0.1:52478)
0812 11:15:00.585015 (+ 2766us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:00.585155 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:32895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2620,"thread_start_us":2398,"threads_started":1}
W20260812 11:15:00.589107 30824 catalog_manager.cc:4584] TS 07037b345dfc454eb0514632d807681d (127.27.109.1:32895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e3d4634db8c4ba79ae8294050d80a3e: Network error: Client connection negotiation failed: client connection to 127.27.109.1:32895: connect: Connection refused (error 111)
I20260812 11:15:00.596426 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31367
I20260812 11:15:00.623908 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30795
2026-08-12T11:15:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7403 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:00 dist-test-slave-7z32 krb5kdc[31816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:00 dist-test-slave-7z32 krb5kdc[31816](info): set up 2 sockets
Aug 12 11:15:00 dist-test-slave-7z32 krb5kdc[31816](info): commencing operation
krb5kdc: starting...
W20260812 11:15:02.023708 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.363s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:02 dist-test-slave-7z32 krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:02.191082 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46261
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:46261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:02.344002 31835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:02.344434 31835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:02.344483 31835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:02.349126 31835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:02.349231 31835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:02.349252 31835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:02.349272 31835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:02.349288 31835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:02.356714 31835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:46261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:02.358399 31835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:02.360011 31835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:02.372437 31841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:02.372440 31842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:02.372767 31835 server_base.cc:1034] running on GCE node
W20260812 11:15:02.372440 31844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:02.373632 31835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:02.375113 31835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:02.376422 31835 hybrid_clock.cc:648] HybridClock initialized: now 1786533302376348 us; error 101 us; skew 500 ppm
Aug 12 11:15:02 dist-test-slave-7z32 krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533302, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:02.379956 31835 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:02.381145 31835 webserver.cc:460] Webserver started at http://127.27.109.62:36475/ using document root <none> and password file <none>
I20260812 11:15:02.381829 31835 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:02.381906 31835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:02.382153 31835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:02.384675 31835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "a982ceed4c614db7b73a5dad4b3cbe18"
format_stamp: "Formatted at 2026-08-12 11:15:02 on dist-test-slave-7z32"
server_key: "58ebab449b15d6cd3382b71f0dce672a"
server_key_iv: "1e15fea4938296b8f7cbb6c289a2b8a1"
server_key_version: "encryptionkey@0"
I20260812 11:15:02.385313 31835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "a982ceed4c614db7b73a5dad4b3cbe18"
format_stamp: "Formatted at 2026-08-12 11:15:02 on dist-test-slave-7z32"
server_key: "58ebab449b15d6cd3382b71f0dce672a"
server_key_iv: "1e15fea4938296b8f7cbb6c289a2b8a1"
server_key_version: "encryptionkey@0"
I20260812 11:15:02.390041 31835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 11:15:02.393278 31854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:02.395285 31835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:02.395515 31835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "a982ceed4c614db7b73a5dad4b3cbe18"
format_stamp: "Formatted at 2026-08-12 11:15:02 on dist-test-slave-7z32"
server_key: "58ebab449b15d6cd3382b71f0dce672a"
server_key_iv: "1e15fea4938296b8f7cbb6c289a2b8a1"
server_key_version: "encryptionkey@0"
I20260812 11:15:02.395649 31835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:02.410027 31835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:02.416257 31835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:02.416577 31835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:02.426319 31835 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:46261
I20260812 11:15:02.426411 31935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:46261 every 8 connection(s)
I20260812 11:15:02.427836 31835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:02.431488 31936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:02.432592 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31835
I20260812 11:15:02.432729 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:02.433084 28084 external_mini_cluster.cc:1428] Setting key 72c1816eb13ffce719a89d3527e44d00
I20260812 11:15:02.439216 31936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18: Bootstrap starting.
I20260812 11:15:02.442309 31936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:02.443423 31936 log.cc:824] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:15:02 dist-test-slave-7z32 krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:02.445930 31936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18: No bootstrap required, opened a new log
I20260812 11:15:02.451622 31936 raft_consensus.cc:348] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46261 } }
I20260812 11:15:02.451951 31936 raft_consensus.cc:374] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:02.452044 31936 raft_consensus.cc:729] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a982ceed4c614db7b73a5dad4b3cbe18, State: Initialized, Role: FOLLOWER
I20260812 11:15:02.452641 31936 consensus_queue.cc:260] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46261 } }
I20260812 11:15:02.452832 31936 raft_consensus.cc:388] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:02.452854 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:02.435122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.27.109.62:46261)
0812 11:15:02.435718 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:02.435742 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:02.435796 (+ 54us) server_negotiation.cc:407] Connection header received
0812 11:15:02.436724 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:02.436749 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:02.437231 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:02.437659 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:02.438724 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:02.440429 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:02.441611 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:02.442439 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:02.442669 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:02.446097 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:02.446104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:02.446123 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:02.446168 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:02.448681 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:02.449356 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:02.449362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:02.449369 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:02.449544 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:02.449861 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:02.449865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:02.449867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:02.450466 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:02.450717 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:02.451209 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 11:15:02.451571 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":174,"threads_started":1}
I20260812 11:15:02.452911 31936 raft_consensus.cc:482] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:02.453013 31936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:02.454124 31936 raft_consensus.cc:504] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46261 } }
I20260812 11:15:02.454561 31936 leader_election.cc:302] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a982ceed4c614db7b73a5dad4b3cbe18; no voters:
I20260812 11:15:02.455052 31936 leader_election.cc:288] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:02.455447 31944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:02.455765 31944 raft_consensus.cc:686] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [term 1 LEADER]: Becoming Leader. State: Replica: a982ceed4c614db7b73a5dad4b3cbe18, State: Running, Role: LEADER
I20260812 11:15:02.456657 31936 sys_catalog.cc:564] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:02.456689 31944 consensus_queue.cc:237] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46261 } }
I20260812 11:15:02.459420 31945 sys_catalog.cc:455] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46261 } } }
I20260812 11:15:02.459681 31947 sys_catalog.cc:455] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a982ceed4c614db7b73a5dad4b3cbe18. Latest consensus state: current_term: 1 leader_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a982ceed4c614db7b73a5dad4b3cbe18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46261 } } }
I20260812 11:15:02.460143 31945 sys_catalog.cc:458] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:02.460158 31947 sys_catalog.cc:458] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18 [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:02.463979 31963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:02.464119 31963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:02.464223 31964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:02.465066 31964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:02.472919 31964 catalog_manager.cc:1347] Generated new cluster ID: a8557349ad954e98ace6237a387b3b3a
I20260812 11:15:02.473060 31964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:02.489692 31964 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:02.491222 31964 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:02.498703 31964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a982ceed4c614db7b73a5dad4b3cbe18: Generated new TSK 0
I20260812 11:15:02.499878 31964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:15:02.539755 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46261
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:02.695931 31972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:02.696333 31972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:02.696388 31972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:02.702272 31972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:02.702395 31972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:02.702508 31972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:02.709713 31972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:02.711189 31972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:02.712554 31972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:02.725898 31985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:02.725871 31987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:02.725922 31983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:02.726518 31972 server_base.cc:1034] running on GCE node
I20260812 11:15:02.727074 31972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:02.727845 31972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:02.729055 31972 hybrid_clock.cc:648] HybridClock initialized: now 1786533302729033 us; error 55 us; skew 500 ppm
Aug 12 11:15:02 dist-test-slave-7z32 krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533302, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:02.732647 31972 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:15:02.733790 31972 webserver.cc:460] Webserver started at http://127.27.109.1:34981/ using document root <none> and password file <none>
I20260812 11:15:02.734476 31972 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:02.734575 31972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:02.734889 31972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:02.737162 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "eb0c21542b6d4c2fa9b2b513bf369d95"
format_stamp: "Formatted at 2026-08-12 11:15:02 on dist-test-slave-7z32"
server_key: "af892e3bc25ac688208f6025354ff114"
server_key_iv: "89d61c2220c236e3e662d1b9fa35fd6b"
server_key_version: "encryptionkey@0"
I20260812 11:15:02.737802 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "eb0c21542b6d4c2fa9b2b513bf369d95"
format_stamp: "Formatted at 2026-08-12 11:15:02 on dist-test-slave-7z32"
server_key: "af892e3bc25ac688208f6025354ff114"
server_key_iv: "89d61c2220c236e3e662d1b9fa35fd6b"
server_key_version: "encryptionkey@0"
I20260812 11:15:02.742455 31972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 11:15:02.745667 31996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:02.747607 31972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:02.747800 31972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "eb0c21542b6d4c2fa9b2b513bf369d95"
format_stamp: "Formatted at 2026-08-12 11:15:02 on dist-test-slave-7z32"
server_key: "af892e3bc25ac688208f6025354ff114"
server_key_iv: "89d61c2220c236e3e662d1b9fa35fd6b"
server_key_version: "encryptionkey@0"
I20260812 11:15:02.747990 31972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:02.769542 31972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:02.773808 31972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:02.774102 31972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:02.774928 31972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:02.775938 31972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:02.776012 31972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:02.776089 31972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:02.776129 31972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:02.787973 31972 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:45397
I20260812 11:15:02.788086 32149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:45397 every 8 connection(s)
I20260812 11:15:02.789376 31972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:02.792070 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31972
I20260812 11:15:02.792243 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:02.792631 28084 external_mini_cluster.cc:1428] Setting key 85a30411e870eca20aa54a0f1f65db3e
Aug 12 11:15:02 dist-test-slave-7z32 krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533302, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:15:02.810654 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:02.792302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:46569 (local address 127.27.109.62:46261)
0812 11:15:02.792554 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:02.792562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:02.794086 (+ 1524us) server_negotiation.cc:407] Connection header received
0812 11:15:02.795225 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:02.795232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:02.795308 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:02.795419 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:02.796882 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:02.797652 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:02.799976 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:02.800294 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:02.800502 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:02.805035 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:02.805040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:02.805042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:02.805086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:02.807266 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:02.808172 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:02.808177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:02.808178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:02.808281 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:02.809057 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:02.809061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:02.809063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:02.809372 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:02.809506 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:02.810235 (+ 729us) server_negotiation.cc:299] Negotiation successful
0812 11:15:02.810444 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:02.812115 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:02.792947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46261 (local address 127.27.109.1:46569)
0812 11:15:02.793865 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:02.793930 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:02.794968 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:02.795609 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:02.795616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:02.796109 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:02.796687 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:02.796702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:02.797891 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:02.797898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:02.799748 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:02.799767 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:02.800576 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:02.800583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:02.800910 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:02.801979 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:02.802013 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:02.804696 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:02.807477 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:02.807498 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:02.807526 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:02.808006 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:02.808406 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:02.808410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:02.808412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:02.808900 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:02.809507 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:02.809521 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:02.809921 (+ 400us) client_negotiation.cc:769] Sending connection context
0812 11:15:02.810167 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 11:15:02.810667 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":166,"threads_started":1}
I20260812 11:15:02.814266 32150 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46261
I20260812 11:15:02.814602 32150 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:02.815359 32150 heartbeater.cc:511] Master 127.27.109.62:46261 requested a full tablet report, sending...
I20260812 11:15:02.817451 31883 ts_manager.cc:194] Registered new tserver with Master: eb0c21542b6d4c2fa9b2b513bf369d95 (127.27.109.1:45397)
I20260812 11:15:02.819078 31883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:46569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:15:02.835763 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46261
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:02.977118 32162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:02.977487 32162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:02.977571 32162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:02.981976 32162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:02.982105 32162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:02.982213 32162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:02.988333 32162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:02.989668 32162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:02.990854 32162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:03.002501 32168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:03.002461 32170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:03.002689 32162 server_base.cc:1034] running on GCE node
W20260812 11:15:03.002534 32176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:03.003403 32162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:03.004154 32162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:03.005357 32162 hybrid_clock.cc:648] HybridClock initialized: now 1786533303005325 us; error 62 us; skew 500 ppm
Aug 12 11:15:03 dist-test-slave-7z32 krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:03.008863 32162 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:15:03.009943 32162 webserver.cc:460] Webserver started at http://127.27.109.2:38817/ using document root <none> and password file <none>
I20260812 11:15:03.010619 32162 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:03.010720 32162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:03.011020 32162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:03.013412 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "51e2a130a4a44730a1e2394a03d9ce3d"
format_stamp: "Formatted at 2026-08-12 11:15:03 on dist-test-slave-7z32"
server_key: "1832700d1932835e9346d05c11393a60"
server_key_iv: "04b75c8a2b2693e56911eb0a574cd7aa"
server_key_version: "encryptionkey@0"
I20260812 11:15:03.014037 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "51e2a130a4a44730a1e2394a03d9ce3d"
format_stamp: "Formatted at 2026-08-12 11:15:03 on dist-test-slave-7z32"
server_key: "1832700d1932835e9346d05c11393a60"
server_key_iv: "04b75c8a2b2693e56911eb0a574cd7aa"
server_key_version: "encryptionkey@0"
I20260812 11:15:03.018734 32162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:15:03.022146 32186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:03.024022 32162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:03.024210 32162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "51e2a130a4a44730a1e2394a03d9ce3d"
format_stamp: "Formatted at 2026-08-12 11:15:03 on dist-test-slave-7z32"
server_key: "1832700d1932835e9346d05c11393a60"
server_key_iv: "04b75c8a2b2693e56911eb0a574cd7aa"
server_key_version: "encryptionkey@0"
I20260812 11:15:03.024363 32162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:03.049077 32162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:03.053197 32162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:03.053434 32162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:03.054093 32162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:03.055081 32162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:03.055151 32162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:03.055223 32162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:03.055272 32162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:03.067303 32162 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43715
I20260812 11:15:03.067420 32351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43715 every 8 connection(s)
I20260812 11:15:03.068388 32162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:03.077617 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 32162
I20260812 11:15:03.077819 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:03.078260 28084 external_mini_cluster.cc:1428] Setting key 32185a273318a974b96cfa763b13104a
Aug 12 11:15:03 dist-test-slave-7z32 krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:03.088308 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:03.070985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:54915 (local address 127.27.109.62:46261)
0812 11:15:03.071175 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:03.071179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:03.072259 (+ 1080us) server_negotiation.cc:407] Connection header received
0812 11:15:03.073982 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:03.073993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:03.074074 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:03.074178 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:03.075836 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.076459 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.078971 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.079381 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.079515 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:03.083886 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:03.083890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:03.083892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:03.083925 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:03.085808 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:03.086457 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:03.086461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:03.086463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:03.086553 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:03.086975 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:03.086978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:03.086980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:03.087214 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:03.087339 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:03.087947 (+ 608us) server_negotiation.cc:299] Negotiation successful
0812 11:15:03.088098 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:15:03.089231 32355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:03.071411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46261 (local address 127.27.109.2:54915)
0812 11:15:03.072011 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:03.072073 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:03.073675 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:03.074414 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:03.074425 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:03.074969 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:03.075632 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:03.075650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.076837 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.076843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:03.078587 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:03.078618 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.079671 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.079677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:03.079961 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:03.080981 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:03.081007 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:03.083564 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:03.085979 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:03.085988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:03.086017 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:03.086335 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:03.086649 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:03.086653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:03.086655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:03.086875 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:03.087340 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:03.087349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:03.087666 (+ 317us) client_negotiation.cc:769] Sending connection context
0812 11:15:03.087883 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 11:15:03.088262 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":128,"threads_started":1}
I20260812 11:15:03.091298 32352 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46261
I20260812 11:15:03.091624 32352 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:03.092236 32352 heartbeater.cc:511] Master 127.27.109.62:46261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:15:03.093693 31883 ts_manager.cc:194] Registered new tserver with Master: 51e2a130a4a44730a1e2394a03d9ce3d (127.27.109.2:43715)
I20260812 11:15:03.094538 31883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:54915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:15:03.119796 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46261
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:03.272818 32359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:03.273238 32359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:03.273295 32359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:03.278040 32359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:03.278158 32359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:03.278272 32359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:03.285100 32359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:03.286759 32359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:03.288113 32359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:03.300531 32367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:03.300532 32368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:03.300702 32370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:03.301092 32359 server_base.cc:1034] running on GCE node
I20260812 11:15:03.301643 32359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:03.302541 32359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:03.303785 32359 hybrid_clock.cc:648] HybridClock initialized: now 1786533303303758 us; error 50 us; skew 500 ppm
Aug 12 11:15:03 dist-test-slave-7z32 krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:03.308022 32359 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:15:03.309476 32359 webserver.cc:460] Webserver started at http://127.27.109.3:46657/ using document root <none> and password file <none>
I20260812 11:15:03.310318 32359 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:03.310403 32359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:03.310658 32359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:03.313990 32359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "05b92800b9ab4d5c8a48b0682d64fb12"
format_stamp: "Formatted at 2026-08-12 11:15:03 on dist-test-slave-7z32"
server_key: "29e0551ca8e71c00925143811fb558c5"
server_key_iv: "f9abec04992fdc12dc1966bb2300c25b"
server_key_version: "encryptionkey@0"
I20260812 11:15:03.314810 32359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "05b92800b9ab4d5c8a48b0682d64fb12"
format_stamp: "Formatted at 2026-08-12 11:15:03 on dist-test-slave-7z32"
server_key: "29e0551ca8e71c00925143811fb558c5"
server_key_iv: "f9abec04992fdc12dc1966bb2300c25b"
server_key_version: "encryptionkey@0"
I20260812 11:15:03.320940 32359 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:03.324906 32383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:03.327189 32359 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:03.327445 32359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "05b92800b9ab4d5c8a48b0682d64fb12"
format_stamp: "Formatted at 2026-08-12 11:15:03 on dist-test-slave-7z32"
server_key: "29e0551ca8e71c00925143811fb558c5"
server_key_iv: "f9abec04992fdc12dc1966bb2300c25b"
server_key_version: "encryptionkey@0"
I20260812 11:15:03.327622 32359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:03.369429 32359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:03.373944 32359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:03.374197 32359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:03.375186 32359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:03.376323 32359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:03.376380 32359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:03.376433 32359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:03.376449 32359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:03.390498 32359 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38665
I20260812 11:15:03.390600 32542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38665 every 8 connection(s)
I20260812 11:15:03.391880 32359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:03.402155 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 32359
I20260812 11:15:03.402336 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:03.402776 28084 external_mini_cluster.cc:1428] Setting key 03ca7f3682cd362ab87b69ab359f72ef
Aug 12 11:15:03 dist-test-slave-7z32 krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:03.421042 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:03.394418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57873 (local address 127.27.109.62:46261)
0812 11:15:03.394760 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:03.394765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:03.396397 (+ 1632us) server_negotiation.cc:407] Connection header received
0812 11:15:03.397550 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:03.397554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:03.397627 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:03.397738 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:03.399493 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.400190 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.402038 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.402343 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.402487 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:03.415490 (+ 13003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:03.415495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:03.415498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:03.415543 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:03.418014 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:03.419170 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:03.419175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:03.419177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:03.419274 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:03.419721 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:03.419725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:03.419727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:03.419958 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:03.420068 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:03.420708 (+ 640us) server_negotiation.cc:299] Negotiation successful
0812 11:15:03.420867 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 11:15:03.422063 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:03.395057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46261 (local address 127.27.109.3:57873)
0812 11:15:03.396147 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:03.396233 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:03.397277 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:03.397966 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:03.397975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:03.398496 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:03.399263 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:03.399283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.400332 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.400336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:03.401847 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:03.401865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:03.402572 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:03.402578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:03.402882 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:03.412726 (+ 9844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:03.412766 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:03.415096 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:03.418327 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:03.418341 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:03.418370 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:03.419002 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:03.419386 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:03.419392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:03.419397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:03.419577 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:03.420096 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:03.420104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:03.420435 (+ 331us) client_negotiation.cc:769] Sending connection context
0812 11:15:03.420649 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 11:15:03.421060 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":178,"threads_started":1}
I20260812 11:15:03.424096 32543 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46261
I20260812 11:15:03.424470 32543 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:03.425148 32543 heartbeater.cc:511] Master 127.27.109.62:46261 requested a full tablet report, sending...
I20260812 11:15:03.427068 31883 ts_manager.cc:194] Registered new tserver with Master: 05b92800b9ab4d5c8a48b0682d64fb12 (127.27.109.3:38665)
I20260812 11:15:03.428047 31883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:57873
I20260812 11:15:03.431777 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:15:03.455400 31939 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:03.453364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.27.109.62:46261)
0812 11:15:03.453556 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:03.453561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:03.453851 (+ 290us) server_negotiation.cc:407] Connection header received
0812 11:15:03.454035 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:03.454038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:03.454092 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:03.454185 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:03.455218 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34158: BlockingRecv error: recv got EOF from 127.0.0.1:34158 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:15:03.456734 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31972
I20260812 11:15:03.478327 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 32162
I20260812 11:15:03.496487 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 32359
I20260812 11:15:03.523731 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31835
2026-08-12T11:15:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2917 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:03 dist-test-slave-7z32 krb5kdc[32559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:03 dist-test-slave-7z32 krb5kdc[32559](info): set up 2 sockets
Aug 12 11:15:03 dist-test-slave-7z32 krb5kdc[32559](info): commencing operation
krb5kdc: starting...
W20260812 11:15:04.967051 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.388s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:05 dist-test-slave-7z32 krb5kdc[32559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:05.152858 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39157
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:05.293380 32578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:05.293833 32578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:05.293891 32578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:05.298919 32578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:05.299022 32578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:05.299044 32578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:05.299062 32578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:05.299079 32578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:05.305173 32578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:05.306412 32578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:05.307565 32578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:05.317878 32587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:05.317878 32591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:05.318127 32578 server_base.cc:1034] running on GCE node
W20260812 11:15:05.318369 32593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:05.319262 32578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:05.320909 32578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:05.322185 32578 hybrid_clock.cc:648] HybridClock initialized: now 1786533305322158 us; error 47 us; skew 500 ppm
Aug 12 11:15:05 dist-test-slave-7z32 krb5kdc[32559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:05.326459 32578 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:05.328513 32578 webserver.cc:460] Webserver started at http://127.27.109.62:43845/ using document root <none> and password file <none>
I20260812 11:15:05.329515 32578 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:05.329651 32578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:05.329962 32578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:05.333359 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "3a7206afb65042a6a5baba1a7a3ea822"
format_stamp: "Formatted at 2026-08-12 11:15:05 on dist-test-slave-7z32"
server_key: "3c3eb983d170b1c96ddc4dc26b6999d5"
server_key_iv: "88c168ebf4bc559ffa8d095230101e58"
server_key_version: "encryptionkey@0"
I20260812 11:15:05.334327 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "3a7206afb65042a6a5baba1a7a3ea822"
format_stamp: "Formatted at 2026-08-12 11:15:05 on dist-test-slave-7z32"
server_key: "3c3eb983d170b1c96ddc4dc26b6999d5"
server_key_iv: "88c168ebf4bc559ffa8d095230101e58"
server_key_version: "encryptionkey@0"
I20260812 11:15:05.341490 32578 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 11:15:05.346530 32605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:05.348644 32578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 11:15:05.348865 32578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "3a7206afb65042a6a5baba1a7a3ea822"
format_stamp: "Formatted at 2026-08-12 11:15:05 on dist-test-slave-7z32"
server_key: "3c3eb983d170b1c96ddc4dc26b6999d5"
server_key_iv: "88c168ebf4bc559ffa8d095230101e58"
server_key_version: "encryptionkey@0"
I20260812 11:15:05.349032 32578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:05.368489 32578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:05.373622 32578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:05.374001 32578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:05.384737 32578 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39157
I20260812 11:15:05.385159 32686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39157 every 8 connection(s)
I20260812 11:15:05.386205 32578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:05.390220 32690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:05.394400 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 32578
I20260812 11:15:05.394563 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:05.394928 28084 external_mini_cluster.cc:1428] Setting key 161493a9fb5a9be347f667e84143b3ff
I20260812 11:15:05.397511 32690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822: Bootstrap starting.
I20260812 11:15:05.400523 32690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:05.401655 32690 log.cc:824] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:05.404877 32690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822: No bootstrap required, opened a new log
Aug 12 11:15:05 dist-test-slave-7z32 krb5kdc[32559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:05.410598 32690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7206afb65042a6a5baba1a7a3ea822" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39157 } }
I20260812 11:15:05.411082 32690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:05.411158 32690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7206afb65042a6a5baba1a7a3ea822, State: Initialized, Role: FOLLOWER
I20260812 11:15:05.411810 32690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7206afb65042a6a5baba1a7a3ea822" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39157 } }
I20260812 11:15:05.411972 32690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:05.412081 32690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:05.412202 32690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:05.413631 32690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7206afb65042a6a5baba1a7a3ea822" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39157 } }
I20260812 11:15:05.414122 32690 leader_election.cc:302] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a7206afb65042a6a5baba1a7a3ea822; no voters:
I20260812 11:15:05.414587 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:05.396997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.27.109.62:39157)
0812 11:15:05.397670 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:05.397699 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:05.397757 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:15:05.398675 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:05.398700 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:05.399288 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:05.399701 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:05.401065 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:05.402400 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:05.403620 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:05.404432 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:05.404658 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:05.407586 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:05.407593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:05.407620 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:05.407664 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:05.410276 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:05.410991 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:05.411001 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:05.411011 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:05.411207 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:05.411635 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:05.411641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:05.411643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:05.412478 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:05.412778 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:05.413074 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 11:15:05.413471 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":191,"threads_started":1}
I20260812 11:15:05.414573 32690 leader_election.cc:288] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:05.414757 32697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:05.415103 32697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7206afb65042a6a5baba1a7a3ea822, State: Running, Role: LEADER
I20260812 11:15:05.415692 32697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7206afb65042a6a5baba1a7a3ea822" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39157 } }
I20260812 11:15:05.416682 32690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:05.417837 32700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a7206afb65042a6a5baba1a7a3ea822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7206afb65042a6a5baba1a7a3ea822" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39157 } } }
I20260812 11:15:05.418038 32700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:05.418462 32698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a7206afb65042a6a5baba1a7a3ea822. Latest consensus state: current_term: 1 leader_uuid: "3a7206afb65042a6a5baba1a7a3ea822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7206afb65042a6a5baba1a7a3ea822" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39157 } } }
I20260812 11:15:05.418630 32698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:05.418936 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:05.419147 32711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:05.422957 32711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:05.431093 32711 catalog_manager.cc:1347] Generated new cluster ID: b1d7dcab3876431da825369bf07e34de
I20260812 11:15:05.431197 32711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:05.444723 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:05.437867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.27.109.62:39157)
0812 11:15:05.437995 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:05.438001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:05.438424 (+ 423us) server_negotiation.cc:407] Connection header received
0812 11:15:05.438619 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:05.438622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:05.438671 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:05.438721 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:05.439598 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:05.440537 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:05.441774 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:05.442175 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:05.442245 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:05.443507 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:05.443511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:05.443514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:05.444184 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:05.444221 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:05.444241 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:05.444303 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:05.444483 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:15:05.444558 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:15:05.458868 32711 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:05.460242 32711 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:05.466092 32711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a7206afb65042a6a5baba1a7a3ea822: Generated new TSK 0
I20260812 11:15:05.467562 32711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:05.479454 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 32578
2026-08-12T11:15:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1936 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:05 dist-test-slave-7z32 krb5kdc[32741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:05 dist-test-slave-7z32 krb5kdc[32741](info): set up 2 sockets
Aug 12 11:15:05 dist-test-slave-7z32 krb5kdc[32741](info): commencing operation
krb5kdc: starting...
W20260812 11:15:06.933429 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.418s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:07 dist-test-slave-7z32 krb5kdc[32741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:07.108620 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33709
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:07.262197 32761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:07.262691 32761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:07.262776 32761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:07.267606 32761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:07.267781 32761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:07.267820 32761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:07.267853 32761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:07.267887 32761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:07.274699 32761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:07.276435 32761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:07.277848 32761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:07.289637 301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:07.289790 302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:07.289642 304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:07.290009 32761 server_base.cc:1034] running on GCE node
I20260812 11:15:07.290903 32761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:07.292282 32761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:07.293527 32761 hybrid_clock.cc:648] HybridClock initialized: now 1786533307293479 us; error 80 us; skew 500 ppm
Aug 12 11:15:07 dist-test-slave-7z32 krb5kdc[32741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:07.297222 32761 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:07.298511 32761 webserver.cc:460] Webserver started at http://127.27.109.62:42851/ using document root <none> and password file <none>
I20260812 11:15:07.299283 32761 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:07.299387 32761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:07.299655 32761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:07.302454 32761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "be3f66f0de2044b6830a7463bb777a76"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "c8ceeb68ed310f36068f7fcf7455ea9f"
server_key_iv: "4f2d2cf8399e692ec87151dece2e4a6e"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.303277 32761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "be3f66f0de2044b6830a7463bb777a76"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "c8ceeb68ed310f36068f7fcf7455ea9f"
server_key_iv: "4f2d2cf8399e692ec87151dece2e4a6e"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.309091 32761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:15:07.313443 312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:07.316118 32761 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:15:07.316362 32761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "be3f66f0de2044b6830a7463bb777a76"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "c8ceeb68ed310f36068f7fcf7455ea9f"
server_key_iv: "4f2d2cf8399e692ec87151dece2e4a6e"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.316613 32761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:07.332983 32761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:07.338192 32761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:07.338505 32761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:07.348230 32761 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:33709
I20260812 11:15:07.348619 387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:33709 every 8 connection(s)
I20260812 11:15:07.349673 32761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:07.350878 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 32761
I20260812 11:15:07.351033 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:07.351444 28084 external_mini_cluster.cc:1428] Setting key e2e4c142c71b251c2ca555e55e7fc0b5
I20260812 11:15:07.355535 388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:07.364207 388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76: Bootstrap starting.
Aug 12 11:15:07 dist-test-slave-7z32 krb5kdc[32741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:07.368050 388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:07.369257 388 log.cc:824] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:07.372699 388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76: No bootstrap required, opened a new log
I20260812 11:15:07.374230 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:07.353713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.27.109.62:33709)
0812 11:15:07.354560 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:07.354589 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:07.354642 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:15:07.355556 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:07.355580 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:07.356109 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:07.356495 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:07.357731 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:07.359243 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:07.360629 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:07.361381 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:07.361567 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:07.366167 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:07.366177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:07.366201 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:07.366359 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:07.369242 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:07.370258 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:07.370349 (+ 91us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:07.370356 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:07.370538 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:07.371292 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:07.371296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:07.371298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:07.371967 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:07.372215 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:07.372664 (+ 449us) server_negotiation.cc:299] Negotiation successful
0812 11:15:07.373080 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":183,"threads_started":1}
I20260812 11:15:07.377887 388 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3f66f0de2044b6830a7463bb777a76" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33709 } }
I20260812 11:15:07.378209 388 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:07.378269 388 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3f66f0de2044b6830a7463bb777a76, State: Initialized, Role: FOLLOWER
I20260812 11:15:07.378744 388 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3f66f0de2044b6830a7463bb777a76" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33709 } }
I20260812 11:15:07.378957 388 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:07.379020 388 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:07.379108 388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:07.380226 388 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3f66f0de2044b6830a7463bb777a76" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33709 } }
I20260812 11:15:07.380614 388 leader_election.cc:302] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3f66f0de2044b6830a7463bb777a76; no voters:
I20260812 11:15:07.380985 388 leader_election.cc:288] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:07.381350 395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:07.381880 395 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [term 1 LEADER]: Becoming Leader. State: Replica: be3f66f0de2044b6830a7463bb777a76, State: Running, Role: LEADER
I20260812 11:15:07.382439 395 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3f66f0de2044b6830a7463bb777a76" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33709 } }
I20260812 11:15:07.382781 388 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:07.385021 396 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3f66f0de2044b6830a7463bb777a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3f66f0de2044b6830a7463bb777a76" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33709 } } }
I20260812 11:15:07.385098 397 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3f66f0de2044b6830a7463bb777a76. Latest consensus state: current_term: 1 leader_uuid: "be3f66f0de2044b6830a7463bb777a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3f66f0de2044b6830a7463bb777a76" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33709 } } }
I20260812 11:15:07.385228 396 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:07.385258 397 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:07.385974 409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:07.389942 409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:07.398464 409 catalog_manager.cc:1347] Generated new cluster ID: 4fd44e60e653417d933ecddc622ab1c8
I20260812 11:15:07.398586 409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:07.413072 409 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:07.414335 409 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:07.428440 409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3f66f0de2044b6830a7463bb777a76: Generated new TSK 0
I20260812 11:15:07.429344 409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:15:07.472105 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33709
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:07.625440 435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:07.625810 435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:07.625864 435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:07.630668 435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:07.630771 435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:07.630859 435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:07.637290 435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:07.638665 435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:07.639909 435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:07.653491 441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:07.653555 440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:07.653551 443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:07.653952 435 server_base.cc:1034] running on GCE node
I20260812 11:15:07.654742 435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:07.655591 435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:07.656963 435 hybrid_clock.cc:648] HybridClock initialized: now 1786533307656921 us; error 71 us; skew 500 ppm
Aug 12 11:15:07 dist-test-slave-7z32 krb5kdc[32741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:07.660809 435 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:15:07.662063 435 webserver.cc:460] Webserver started at http://127.27.109.1:45997/ using document root <none> and password file <none>
I20260812 11:15:07.662737 435 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:07.662866 435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:07.663160 435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:07.665978 435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "359404de7d8244f2ab068ccecb64f02e"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "2ced1818e683ed48165b63f321969846"
server_key_iv: "101c773b137959854610fbe0aab8d3c1"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.666816 435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "359404de7d8244f2ab068ccecb64f02e"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "2ced1818e683ed48165b63f321969846"
server_key_iv: "101c773b137959854610fbe0aab8d3c1"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.672241 435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:15:07.676056 451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:07.679531 435 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:07.679791 435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "359404de7d8244f2ab068ccecb64f02e"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "2ced1818e683ed48165b63f321969846"
server_key_iv: "101c773b137959854610fbe0aab8d3c1"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.679976 435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:07.693990 435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:07.698535 435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:07.698900 435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:07.699743 435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:07.700901 435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:07.700989 435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:07.701073 435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:07.701128 435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:07.714697 435 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37351
I20260812 11:15:07.714732 615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37351 every 8 connection(s)
I20260812 11:15:07.716019 435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:07.725065 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 435
I20260812 11:15:07.725219 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:07.725793 28084 external_mini_cluster.cc:1428] Setting key 06c73232cca9c7623c7149d90bbcb26c
Aug 12 11:15:07 dist-test-slave-7z32 krb5kdc[32741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:07.740051 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:07.719563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:55569 (local address 127.27.109.62:33709)
0812 11:15:07.719824 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:07.719832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:07.721216 (+ 1384us) server_negotiation.cc:407] Connection header received
0812 11:15:07.723386 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:07.723391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:07.723560 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:07.723692 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:07.725585 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:07.726437 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:07.728402 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:07.729019 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:07.729193 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:07.733740 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:07.733749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:07.733755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:07.733811 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:07.736561 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:07.737507 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:07.737513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:07.737518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:07.737664 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:07.738182 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:07.738186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:07.738187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:07.738526 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:07.738658 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:07.739603 (+ 945us) server_negotiation.cc:299] Negotiation successful
0812 11:15:07.739795 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:15:07.741365 618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:07.719761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33709 (local address 127.27.109.1:55569)
0812 11:15:07.720889 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:07.720978 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:07.722976 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:07.723979 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:07.723990 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:07.724561 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:07.725356 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:07.725376 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:07.726681 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:07.726687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:07.728128 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:07.728148 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:07.729456 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:07.729460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:07.729771 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:07.730747 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:07.730771 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:07.733391 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:07.736809 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:07.736819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:07.736843 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:07.737314 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:07.737803 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:07.737806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:07.737808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:07.738029 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:07.738705 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:07.738713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:07.739228 (+ 515us) client_negotiation.cc:769] Sending connection context
0812 11:15:07.739469 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 11:15:07.739939 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":338,"threads_started":1}
I20260812 11:15:07.743556 616 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33709
I20260812 11:15:07.744030 616 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:07.744884 616 heartbeater.cc:511] Master 127.27.109.62:33709 requested a full tablet report, sending...
I20260812 11:15:07.747958 335 ts_manager.cc:194] Registered new tserver with Master: 359404de7d8244f2ab068ccecb64f02e (127.27.109.1:37351)
I20260812 11:15:07.750206 335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:55569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:15:07.771893 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33709
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:07.904198 623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:07.904738 623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:07.904812 623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:07.909845 623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:07.909960 623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:07.910033 623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:07.917063 623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:07.918378 623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:07.919813 623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:07.933930 631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:07.934027 639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:07.933777 632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:07.934456 623 server_base.cc:1034] running on GCE node
I20260812 11:15:07.935240 623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:07.936262 623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:07.937479 623 hybrid_clock.cc:648] HybridClock initialized: now 1786533307937430 us; error 69 us; skew 500 ppm
Aug 12 11:15:07 dist-test-slave-7z32 krb5kdc[32741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:07.941634 623 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:15:07.943238 623 webserver.cc:460] Webserver started at http://127.27.109.2:32895/ using document root <none> and password file <none>
I20260812 11:15:07.944195 623 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:07.944294 623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:07.944571 623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:07.947959 623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "1a660fe2c17a49fa919d4ea04b7244e1"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "7f850be49f1008788e847c82d13e708e"
server_key_iv: "bf503e0756c3605b7f27c4ae879d014e"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.948844 623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "1a660fe2c17a49fa919d4ea04b7244e1"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "7f850be49f1008788e847c82d13e708e"
server_key_iv: "bf503e0756c3605b7f27c4ae879d014e"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.954771 623 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:07.959406 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:07.961325 623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 11:15:07.961509 623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "1a660fe2c17a49fa919d4ea04b7244e1"
format_stamp: "Formatted at 2026-08-12 11:15:07 on dist-test-slave-7z32"
server_key: "7f850be49f1008788e847c82d13e708e"
server_key_iv: "bf503e0756c3605b7f27c4ae879d014e"
server_key_version: "encryptionkey@0"
I20260812 11:15:07.961742 623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:07.997653 623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:08.001888 623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:08.002099 623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:08.002825 623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:08.004045 623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:08.004107 623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:08.004156 623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:08.004174 623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:08.018088 623 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:33847
I20260812 11:15:08.018188 816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:33847 every 8 connection(s)
I20260812 11:15:08.019368 623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:08.023730 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 623
I20260812 11:15:08.023916 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:08.024328 28084 external_mini_cluster.cc:1428] Setting key 55af21ceb53a2252a4ae56a8fb145aa4
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[32741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:15:08.043272 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.022413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:40463 (local address 127.27.109.62:33709)
0812 11:15:08.022708 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:08.022713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:08.024187 (+ 1474us) server_negotiation.cc:407] Connection header received
0812 11:15:08.026890 (+ 2703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:08.026895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:08.026986 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:08.027079 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:08.029645 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.030525 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.033144 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.033430 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.033570 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:08.038093 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:08.038097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:08.038099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:08.038134 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:08.040171 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:08.041086 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:08.041092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:08.041094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:08.041199 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:08.041719 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:08.041722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:08.041724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:08.041961 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:08.042097 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:08.042887 (+ 790us) server_negotiation.cc:299] Negotiation successful
0812 11:15:08.043060 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:15:08.044602 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.022927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33709 (local address 127.27.109.2:40463)
0812 11:15:08.023840 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:08.023952 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:08.026315 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:08.027515 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:08.027528 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:08.028334 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:08.029311 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:08.029344 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.030864 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.030871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:08.032830 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:08.032858 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.033705 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.033721 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:08.034063 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:08.035160 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:08.035196 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:08.037795 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:08.040478 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:08.040488 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:08.040520 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:08.040947 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:08.041403 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:08.041407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:08.041409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:08.041599 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:08.042098 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:08.042108 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:08.042454 (+ 346us) client_negotiation.cc:769] Sending connection context
0812 11:15:08.042738 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 11:15:08.043203 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":190,"threads_started":1}
I20260812 11:15:08.046936 818 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33709
I20260812 11:15:08.047533 818 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:08.048424 818 heartbeater.cc:511] Master 127.27.109.62:33709 requested a full tablet report, sending...
I20260812 11:15:08.050357 335 ts_manager.cc:194] Registered new tserver with Master: 1a660fe2c17a49fa919d4ea04b7244e1 (127.27.109.2:33847)
I20260812 11:15:08.051278 335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:40463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:15:08.069343 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33709
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:08.216125 824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:08.216568 824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:08.216648 824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:08.221235 824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:08.221318 824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:08.221382 824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:08.227456 824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:08.228761 824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:08.230027 824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:08.243096 834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:08.243453 824 server_base.cc:1034] running on GCE node
W20260812 11:15:08.243258 830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:08.243098 831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:08.244516 824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:08.245391 824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:08.248826 824 hybrid_clock.cc:648] HybridClock initialized: now 1786533308248760 us; error 84 us; skew 500 ppm
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[32741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533308, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:08.252970 824 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:15:08.254407 824 webserver.cc:460] Webserver started at http://127.27.109.3:45013/ using document root <none> and password file <none>
I20260812 11:15:08.255298 824 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:08.255429 824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:08.255752 824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:08.258348 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "e4e049f0d674446a90933f2612946740"
format_stamp: "Formatted at 2026-08-12 11:15:08 on dist-test-slave-7z32"
server_key: "8c675da656fa1dbd85f7d8f82d1ff916"
server_key_iv: "de754c532a5078955482d3ce6974ecf2"
server_key_version: "encryptionkey@0"
I20260812 11:15:08.259143 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "e4e049f0d674446a90933f2612946740"
format_stamp: "Formatted at 2026-08-12 11:15:08 on dist-test-slave-7z32"
server_key: "8c675da656fa1dbd85f7d8f82d1ff916"
server_key_iv: "de754c532a5078955482d3ce6974ecf2"
server_key_version: "encryptionkey@0"
I20260812 11:15:08.264662 824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:15:08.268759 844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:08.270798 824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:08.271075 824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "e4e049f0d674446a90933f2612946740"
format_stamp: "Formatted at 2026-08-12 11:15:08 on dist-test-slave-7z32"
server_key: "8c675da656fa1dbd85f7d8f82d1ff916"
server_key_iv: "de754c532a5078955482d3ce6974ecf2"
server_key_version: "encryptionkey@0"
I20260812 11:15:08.271234 824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:08.285883 824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:08.290179 824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:08.290458 824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:08.291301 824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:08.292368 824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:08.292454 824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:08.292536 824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:08.292582 824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:08.304754 824 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35007
I20260812 11:15:08.304872 1003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35007 every 8 connection(s)
I20260812 11:15:08.305980 824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:08.311618 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 824
I20260812 11:15:08.311759 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:08.312183 28084 external_mini_cluster.cc:1428] Setting key a64d778c7cd03797afddf2d20735d33c
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[32741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533308, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:08.325816 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.308957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:35925 (local address 127.27.109.62:33709)
0812 11:15:08.309140 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:08.309145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:08.310300 (+ 1155us) server_negotiation.cc:407] Connection header received
0812 11:15:08.312440 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:08.312445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:08.312531 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:08.312722 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:08.314778 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.315624 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.317094 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.317345 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.317471 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:08.321321 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:08.321324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:08.321327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:08.321358 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:08.323247 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:08.324039 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:08.324043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:08.324045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:08.324138 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:08.324523 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:08.324527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:08.324529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:08.324754 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:08.324878 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:08.325473 (+ 595us) server_negotiation.cc:299] Negotiation successful
0812 11:15:08.325632 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:15:08.326994 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.309276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33709 (local address 127.27.109.3:35925)
0812 11:15:08.310005 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:08.310082 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:08.312010 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:08.313079 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:08.313095 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:08.313775 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:08.314547 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:08.314575 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.315791 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.315795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:08.316946 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:08.316959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.317528 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.317533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:08.317728 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:08.318575 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:08.318595 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:08.321048 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:08.323511 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:08.323520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:08.323541 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:08.323911 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:08.324239 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:08.324243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:08.324246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:08.324415 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:08.324877 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:08.324886 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:08.325199 (+ 313us) client_negotiation.cc:769] Sending connection context
0812 11:15:08.325427 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 11:15:08.325849 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":173,"threads_started":1}
I20260812 11:15:08.329005 1004 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33709
I20260812 11:15:08.329394 1004 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:08.330214 1004 heartbeater.cc:511] Master 127.27.109.62:33709 requested a full tablet report, sending...
I20260812 11:15:08.331920 333 ts_manager.cc:194] Registered new tserver with Master: e4e049f0d674446a90933f2612946740 (127.27.109.3:35007)
I20260812 11:15:08.332711 333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:35925
I20260812 11:15:08.341838 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[32741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533308, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[32741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533308, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:08.395463 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.380071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.27.109.62:33709)
0812 11:15:08.380342 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:08.380349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:08.380794 (+ 445us) server_negotiation.cc:407] Connection header received
0812 11:15:08.381053 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:08.381058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:08.381133 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:08.381260 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:08.382793 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.383701 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.385684 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:08.386104 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:08.386293 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:08.390640 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:08.390646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:08.390649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:08.390702 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:08.393056 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:08.393802 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:08.393812 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:08.393813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:08.393919 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:08.394248 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:08.394252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:08.394253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:08.394591 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:08.394771 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:08.395076 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 11:15:08.395240 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260812 11:15:08.396051 333 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52866
I20260812 11:15:08.397348 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 435
I20260812 11:15:08.428511 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 623
I20260812 11:15:08.443594 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 824
I20260812 11:15:08.460435 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 32761
2026-08-12T11:15:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2994 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[1020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[1020](info): set up 2 sockets
Aug 12 11:15:08 dist-test-slave-7z32 krb5kdc[1020](info): commencing operation
krb5kdc: starting...
W20260812 11:15:09.860199 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.350s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:09 dist-test-slave-7z32 krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:10.043161 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42487
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:10.200996 1038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:10.201526 1038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:10.201637 1038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:10.206444 1038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:10.206597 1038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:10.206637 1038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:10.206672 1038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:10.206722 1038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:10.213546 1038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:10.215301 1038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:10.216923 1038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:10.229799 1045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:10.230135 1044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:10.229799 1047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:10.230384 1038 server_base.cc:1034] running on GCE node
I20260812 11:15:10.231436 1038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:10.233156 1038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:10.234560 1038 hybrid_clock.cc:648] HybridClock initialized: now 1786533310234525 us; error 47 us; skew 500 ppm
Aug 12 11:15:10 dist-test-slave-7z32 krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:10.239404 1038 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:10.241120 1038 webserver.cc:460] Webserver started at http://127.27.109.62:39483/ using document root <none> and password file <none>
I20260812 11:15:10.242192 1038 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:10.242506 1038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:10.242973 1038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:10.246416 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "faaa1bec3f53493a9a5945dda29e6518"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "e841350be8ab4645f68e48c184290ebb"
server_key_iv: "a09c4b09b51cb767846e8f079b3a575e"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.247407 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "faaa1bec3f53493a9a5945dda29e6518"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "e841350be8ab4645f68e48c184290ebb"
server_key_iv: "a09c4b09b51cb767846e8f079b3a575e"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.255656 1038 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 11:15:10.259244 1061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:10.261400 1038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:10.261687 1038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "faaa1bec3f53493a9a5945dda29e6518"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "e841350be8ab4645f68e48c184290ebb"
server_key_iv: "a09c4b09b51cb767846e8f079b3a575e"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.261891 1038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:10.279182 1038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:10.284480 1038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:10.284807 1038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:10.296030 1038 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:42487
I20260812 11:15:10.296147 1135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:42487 every 8 connection(s)
I20260812 11:15:10.297412 1038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:10.302114 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:10.306026 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 1038
I20260812 11:15:10.306219 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:10.306597 28084 external_mini_cluster.cc:1428] Setting key c26b1f21c2816c6fdca462ebae032491
I20260812 11:15:10.312283 1137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518: Bootstrap starting.
I20260812 11:15:10.316893 1137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:10.318377 1137 log.cc:824] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:15:10 dist-test-slave-7z32 krb5kdc[1020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:10.321521 1137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518: No bootstrap required, opened a new log
I20260812 11:15:10.327629 1137 raft_consensus.cc:348] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa1bec3f53493a9a5945dda29e6518" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42487 } }
I20260812 11:15:10.328022 1137 raft_consensus.cc:374] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:10.328086 1137 raft_consensus.cc:729] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faaa1bec3f53493a9a5945dda29e6518, State: Initialized, Role: FOLLOWER
I20260812 11:15:10.328754 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:10.308940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.27.109.62:42487)
0812 11:15:10.309600 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:10.309627 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:10.309701 (+ 74us) server_negotiation.cc:407] Connection header received
0812 11:15:10.310794 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:10.310880 (+ 86us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:10.311492 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:10.312026 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:10.313172 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:10.315065 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:10.316690 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:10.317798 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:10.318041 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:10.321308 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:10.321316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:10.321340 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:10.321379 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:10.324184 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:10.324864 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:10.324874 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:10.324887 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:10.325122 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:10.325512 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:10.325516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:10.325517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:10.326264 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:10.326529 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:10.326955 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 11:15:10.327389 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":190,"threads_started":1}
I20260812 11:15:10.328701 1137 consensus_queue.cc:260] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa1bec3f53493a9a5945dda29e6518" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42487 } }
I20260812 11:15:10.328886 1137 raft_consensus.cc:388] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:10.328953 1137 raft_consensus.cc:482] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:10.329092 1137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:10.330752 1137 raft_consensus.cc:504] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa1bec3f53493a9a5945dda29e6518" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42487 } }
I20260812 11:15:10.331272 1137 leader_election.cc:302] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faaa1bec3f53493a9a5945dda29e6518; no voters:
I20260812 11:15:10.331784 1137 leader_election.cc:288] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:10.332007 1147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:10.332458 1147 raft_consensus.cc:686] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [term 1 LEADER]: Becoming Leader. State: Replica: faaa1bec3f53493a9a5945dda29e6518, State: Running, Role: LEADER
I20260812 11:15:10.332882 1147 consensus_queue.cc:237] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa1bec3f53493a9a5945dda29e6518" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42487 } }
I20260812 11:15:10.333441 1137 sys_catalog.cc:564] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:10.335144 1150 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faaa1bec3f53493a9a5945dda29e6518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa1bec3f53493a9a5945dda29e6518" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42487 } } }
I20260812 11:15:10.335296 1148 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faaa1bec3f53493a9a5945dda29e6518. Latest consensus state: current_term: 1 leader_uuid: "faaa1bec3f53493a9a5945dda29e6518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa1bec3f53493a9a5945dda29e6518" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42487 } } }
I20260812 11:15:10.335340 1150 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:10.335381 1148 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:10.335934 1159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:10.339630 1159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:10.346870 1159 catalog_manager.cc:1347] Generated new cluster ID: a7fc49ada06747dda76d6ba3ac9a4cb4
I20260812 11:15:10.347021 1159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:10.377946 1159 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:10.379746 1159 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:10.389082 1159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faaa1bec3f53493a9a5945dda29e6518: Generated new TSK 0
I20260812 11:15:10.390172 1159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:15:10.444176 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42487
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:10.619630 1175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:10.619966 1175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:10.620014 1175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:10.624532 1175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:10.624621 1175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:10.624686 1175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:10.630481 1175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:10.631865 1175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:10.633016 1175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:10.645256 1175 server_base.cc:1034] running on GCE node
W20260812 11:15:10.645411 1186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:10.645632 1185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:10.645941 1192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:10.646466 1175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:10.647624 1175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:10.648988 1175 hybrid_clock.cc:648] HybridClock initialized: now 1786533310648932 us; error 81 us; skew 500 ppm
Aug 12 11:15:10 dist-test-slave-7z32 krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:10.652511 1175 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:15:10.653748 1175 webserver.cc:460] Webserver started at http://127.27.109.1:40137/ using document root <none> and password file <none>
I20260812 11:15:10.654408 1175 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:10.654481 1175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:10.654671 1175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:10.657065 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "5cb617856d1f4505982d83650b5f5cff"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "c9e6c543ae70a3561f1d944da0163591"
server_key_iv: "800c3336575642b69ce8abf71e534757"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.657665 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "5cb617856d1f4505982d83650b5f5cff"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "c9e6c543ae70a3561f1d944da0163591"
server_key_iv: "800c3336575642b69ce8abf71e534757"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.662634 1175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 11:15:10.666617 1202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:10.668849 1175 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:10.669046 1175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "5cb617856d1f4505982d83650b5f5cff"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "c9e6c543ae70a3561f1d944da0163591"
server_key_iv: "800c3336575642b69ce8abf71e534757"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.669202 1175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:10.695106 1175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:10.699651 1175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:10.699932 1175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:10.700802 1175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:10.702003 1175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:10.702095 1175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:10.702176 1175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:10.702227 1175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:10.716590 1175 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:41173
I20260812 11:15:10.716714 1366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:41173 every 8 connection(s)
I20260812 11:15:10.717743 1175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:10.718739 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 1175
I20260812 11:15:10.718936 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:10.719316 28084 external_mini_cluster.cc:1428] Setting key e3ccef69845a897c3537be678a3c1fbb
Aug 12 11:15:10 dist-test-slave-7z32 krb5kdc[1020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:10.739032 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:10.721126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:36863 (local address 127.27.109.62:42487)
0812 11:15:10.721343 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:10.721350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:10.722859 (+ 1509us) server_negotiation.cc:407] Connection header received
0812 11:15:10.724776 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:10.724780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:10.724854 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:10.724948 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:10.726674 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:10.727395 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:10.728798 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:10.729072 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:10.729204 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:10.732995 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:10.732999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:10.733001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:10.733046 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:10.735208 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:10.736182 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:10.736186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:10.736188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:10.736326 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:10.737103 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:10.737110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:10.737112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:10.737549 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:10.737729 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:10.738490 (+ 761us) server_negotiation.cc:299] Negotiation successful
0812 11:15:10.738713 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:15:10.740546 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:10.721699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42487 (local address 127.27.109.1:36863)
0812 11:15:10.722503 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:10.722655 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:10.724385 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:10.725201 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:10.725212 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:10.725738 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:10.726446 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:10.726464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:10.727572 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:10.727576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:10.728646 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:10.728658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:10.729275 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:10.729279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:10.729505 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:10.730404 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:10.730425 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:10.732745 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:10.735432 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:10.735440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:10.735464 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:10.735894 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:10.736485 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:10.736492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:10.736497 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:10.736759 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:10.737730 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:10.737738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:10.738138 (+ 400us) client_negotiation.cc:769] Sending connection context
0812 11:15:10.738381 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 11:15:10.738939 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":245,"threads_started":1}
I20260812 11:15:10.742663 1367 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42487
I20260812 11:15:10.743119 1367 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:10.743971 1367 heartbeater.cc:511] Master 127.27.109.62:42487 requested a full tablet report, sending...
I20260812 11:15:10.746277 1085 ts_manager.cc:194] Registered new tserver with Master: 5cb617856d1f4505982d83650b5f5cff (127.27.109.1:41173)
I20260812 11:15:10.747921 1085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:36863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:15:10.764448 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42487
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:10.904742 1374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:10.905126 1374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:10.905180 1374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:10.910089 1374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:10.910192 1374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:10.910256 1374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:10.916497 1374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:10.917784 1374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:10.918977 1374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:10.931496 1383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:10.931486 1387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:10.931571 1374 server_base.cc:1034] running on GCE node
W20260812 11:15:10.931496 1385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:10.932596 1374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:10.933585 1374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:10.934953 1374 hybrid_clock.cc:648] HybridClock initialized: now 1786533310934910 us; error 92 us; skew 500 ppm
Aug 12 11:15:10 dist-test-slave-7z32 krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:10.939042 1374 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:15:10.940454 1374 webserver.cc:460] Webserver started at http://127.27.109.2:40827/ using document root <none> and password file <none>
I20260812 11:15:10.941439 1374 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:10.941587 1374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:10.941916 1374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:10.945060 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "d61146628f12427ba3499fe36c0bb40b"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "54265194d194821ff9ed4e6058adec14"
server_key_iv: "ef354d847dcb5a3080f8896134f766fe"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.945887 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "d61146628f12427ba3499fe36c0bb40b"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "54265194d194821ff9ed4e6058adec14"
server_key_iv: "ef354d847dcb5a3080f8896134f766fe"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.951803 1374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:15:10.955955 1396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:10.958019 1374 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:10.958269 1374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "d61146628f12427ba3499fe36c0bb40b"
format_stamp: "Formatted at 2026-08-12 11:15:10 on dist-test-slave-7z32"
server_key: "54265194d194821ff9ed4e6058adec14"
server_key_iv: "ef354d847dcb5a3080f8896134f766fe"
server_key_version: "encryptionkey@0"
I20260812 11:15:10.958452 1374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:10.972836 1374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:10.977931 1374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:10.978334 1374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:10.979353 1374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:10.980643 1374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:10.980736 1374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:10.980823 1374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:10.980868 1374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:10.999768 1374 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44925
I20260812 11:15:10.999891 1564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44925 every 8 connection(s)
I20260812 11:15:11.002286 1374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:11.007803 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 1374
I20260812 11:15:11.008419 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:11.009080 28084 external_mini_cluster.cc:1428] Setting key 7e0c7bbefbbea835d3c7644a7287c63e
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:15:11.032418 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:11.005550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:55763 (local address 127.27.109.62:42487)
0812 11:15:11.005811 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:11.005816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:11.008001 (+ 2185us) server_negotiation.cc:407] Connection header received
0812 11:15:11.010343 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:11.010349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:11.010442 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:11.010612 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:11.014098 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.015061 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.016865 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.017359 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.017560 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:11.022623 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:11.022632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:11.022637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:11.022710 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:11.026312 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.027957 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.027962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.027963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.028120 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.029421 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.029424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.029426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.029707 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:11.030046 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:11.032070 (+ 2024us) server_negotiation.cc:299] Negotiation successful
0812 11:15:11.032252 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 11:15:11.034667 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:11.006080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42487 (local address 127.27.109.2:55763)
0812 11:15:11.007740 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:11.007825 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:11.009643 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:11.011238 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:11.011254 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:11.012369 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:11.013538 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:11.013586 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.015303 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.015310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:11.016634 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:11.016652 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.017691 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.017696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:11.017940 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:11.019015 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:11.019040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:11.021928 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:11.026574 (+ 4646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:11.026586 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:11.026613 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:11.027663 (+ 1050us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:11.028369 (+ 706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:11.028373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:11.028375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:11.029153 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:11.030221 (+ 1068us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:11.030240 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:11.031462 (+ 1222us) client_negotiation.cc:769] Sending connection context
0812 11:15:11.031902 (+ 440us) client_negotiation.cc:240] Negotiation successful
0812 11:15:11.032906 (+ 1004us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"thread_start_us":915,"threads_started":1}
I20260812 11:15:11.036947 1565 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42487
I20260812 11:15:11.037426 1565 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:11.038312 1565 heartbeater.cc:511] Master 127.27.109.62:42487 requested a full tablet report, sending...
I20260812 11:15:11.040185 1085 ts_manager.cc:194] Registered new tserver with Master: d61146628f12427ba3499fe36c0bb40b (127.27.109.2:44925)
I20260812 11:15:11.041224 1085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:55763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:15:11.061296 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42487
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:11.195921 1573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:11.196306 1573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:11.196359 1573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:11.201726 1573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:11.201860 1573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:11.202317 1573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:11.211421 1573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:11.212807 1573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:11.213999 1573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:11.226253 1580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:11.226356 1581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:11.226389 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:11.226861 1573 server_base.cc:1034] running on GCE node
I20260812 11:15:11.227566 1573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:11.228382 1573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:11.229624 1573 hybrid_clock.cc:648] HybridClock initialized: now 1786533311229595 us; error 50 us; skew 500 ppm
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:11.234007 1573 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:15:11.235863 1573 webserver.cc:460] Webserver started at http://127.27.109.3:38455/ using document root <none> and password file <none>
I20260812 11:15:11.236581 1573 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:11.236670 1573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:11.236882 1573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:11.239801 1573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "c5861d09f41641ef9d9a01f29ee90374"
format_stamp: "Formatted at 2026-08-12 11:15:11 on dist-test-slave-7z32"
server_key: "4ffb83e842aa3f7db73ae8e8332ba9be"
server_key_iv: "a2d517eaed17d59b3571495f5140f7c2"
server_key_version: "encryptionkey@0"
I20260812 11:15:11.240537 1573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "c5861d09f41641ef9d9a01f29ee90374"
format_stamp: "Formatted at 2026-08-12 11:15:11 on dist-test-slave-7z32"
server_key: "4ffb83e842aa3f7db73ae8e8332ba9be"
server_key_iv: "a2d517eaed17d59b3571495f5140f7c2"
server_key_version: "encryptionkey@0"
I20260812 11:15:11.245831 1573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:11.249389 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:11.251497 1573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:15:11.251688 1573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "c5861d09f41641ef9d9a01f29ee90374"
format_stamp: "Formatted at 2026-08-12 11:15:11 on dist-test-slave-7z32"
server_key: "4ffb83e842aa3f7db73ae8e8332ba9be"
server_key_iv: "a2d517eaed17d59b3571495f5140f7c2"
server_key_version: "encryptionkey@0"
I20260812 11:15:11.251929 1573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:11.270308 1573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:11.275305 1573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:11.275594 1573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:11.276348 1573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:11.277398 1573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:11.277494 1573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:11.277580 1573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:11.277632 1573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:11.290323 1573 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:33759
I20260812 11:15:11.290407 1756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:33759 every 8 connection(s)
I20260812 11:15:11.291574 1573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:11.294543 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 1573
I20260812 11:15:11.294749 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:11.295269 28084 external_mini_cluster.cc:1428] Setting key 65d1a9c2688015579d10c2c219018394
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:11.313898 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:11.294318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:56667 (local address 127.27.109.62:42487)
0812 11:15:11.294658 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:11.294666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:11.295937 (+ 1271us) server_negotiation.cc:407] Connection header received
0812 11:15:11.298478 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:11.298482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:11.298551 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:11.298668 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:11.300913 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.301544 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.303762 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.304062 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.304172 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:11.308404 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:11.308409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:11.308411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:11.308452 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:11.310316 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.311342 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.311345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.311347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.311511 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.312081 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.312083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.312084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.312329 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:11.312438 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:11.313396 (+ 958us) server_negotiation.cc:299] Negotiation successful
0812 11:15:11.313650 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 11:15:11.315382 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:11.294748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42487 (local address 127.27.109.3:56667)
0812 11:15:11.295592 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:11.295684 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:11.298042 (+ 2358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:11.299019 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:11.299031 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:11.299719 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:11.300620 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:11.300652 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.301738 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.301742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:11.303499 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:11.303516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.304293 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.304298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:11.304593 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:11.305482 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:11.305510 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:11.307974 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:11.310494 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:11.310504 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:11.310531 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:11.311122 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:11.311643 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:11.311647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:11.311650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:11.311908 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:11.312464 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:11.312474 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:11.312985 (+ 511us) client_negotiation.cc:769] Sending connection context
0812 11:15:11.313264 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 11:15:11.313829 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":172,"threads_started":1}
I20260812 11:15:11.317154 1757 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42487
I20260812 11:15:11.317467 1757 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:11.318200 1757 heartbeater.cc:511] Master 127.27.109.62:42487 requested a full tablet report, sending...
I20260812 11:15:11.319736 1085 ts_manager.cc:194] Registered new tserver with Master: c5861d09f41641ef9d9a01f29ee90374 (127.27.109.3:33759)
I20260812 11:15:11.320454 1085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:56667
I20260812 11:15:11.329692 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:15:11.367913 1766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:11.354237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.27.109.1:41173)
0812 11:15:11.354687 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:11.354698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:11.354739 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:15:11.355046 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:11.355055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:11.355371 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:11.355619 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:11.356651 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.357702 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.358792 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.359355 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.359508 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:11.363069 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:11.363085 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:11.363103 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:11.363151 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:11.365241 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.365944 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.365951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.365952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.366052 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.366455 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.366458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.366459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.366886 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:11.367113 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:11.367449 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 11:15:11.367613 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:11.385571 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:11.373954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.27.109.62:42487)
0812 11:15:11.374178 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:11.374185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:11.374399 (+ 214us) server_negotiation.cc:407] Connection header received
0812 11:15:11.374596 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:11.374599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:11.374659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:11.374748 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:11.375700 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.376256 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.377127 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:11.377390 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:11.377553 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:11.380706 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:11.380710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:11.380712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:11.380745 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:11.382540 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.383305 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.383315 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.383321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.383544 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:11.384090 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:11.384095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:11.384099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:11.384607 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:11.384766 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:11.385103 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 11:15:11.385346 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 11:15:11.386166 1085 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33378
I20260812 11:15:11.387331 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 1175
I20260812 11:15:11.413653 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 1374
I20260812 11:15:11.433199 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 1573
I20260812 11:15:11.443724 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 1038
2026-08-12T11:15:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2987 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1773](info): set up 2 sockets
Aug 12 11:15:11 dist-test-slave-7z32 krb5kdc[1773](info): commencing operation
krb5kdc: starting...
W20260812 11:15:12.817147 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:12 dist-test-slave-7z32 krb5kdc[1773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:12.995675 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37259
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:13.118613 1791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:13.119015 1791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:13.119099 1791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:13.123701 1791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:13.123804 1791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:13.123842 1791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:13.123883 1791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:13.123924 1791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:13.129729 1791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:13.131095 1791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:13.132575 1791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:13.142428 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.142472 1803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.142488 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.142927 1791 server_base.cc:1034] running on GCE node
I20260812 11:15:13.143644 1791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:13.145133 1791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:13.146355 1791 hybrid_clock.cc:648] HybridClock initialized: now 1786533313146322 us; error 55 us; skew 500 ppm
Aug 12 11:15:13 dist-test-slave-7z32 krb5kdc[1773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:13.149812 1791 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:13.150743 1791 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:13.150883 1791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:13.151098 1791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:13.153326 1791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "709247b898a249438c9da96230b1a393"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "823fbdde26a13e1986aceb889b02c9ee"
server_key_iv: "35737a036cbc479c94de843fb2c5c829"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.153926 1791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "709247b898a249438c9da96230b1a393"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "823fbdde26a13e1986aceb889b02c9ee"
server_key_iv: "35737a036cbc479c94de843fb2c5c829"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.158380 1791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 11:15:13.161455 1813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.163398 1791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:13.163563 1791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "709247b898a249438c9da96230b1a393"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "823fbdde26a13e1986aceb889b02c9ee"
server_key_iv: "35737a036cbc479c94de843fb2c5c829"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.163692 1791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:13.179141 1791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:13.183334 1791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:13.183539 1791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:13.192054 1791 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:37259
I20260812 11:15:13.192466 1894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:37259 every 8 connection(s)
I20260812 11:15:13.192893 1791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:13.196915 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 1791
I20260812 11:15:13.197163 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:13.197371 1895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:13.197587 28084 external_mini_cluster.cc:1428] Setting key a81597f40c8b1433ac86c1a2b128e3c4
I20260812 11:15:13.207353 1895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393: Bootstrap starting.
Aug 12 11:15:13 dist-test-slave-7z32 krb5kdc[1773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:13.211570 1895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:13.212697 1895 log.cc:824] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:13.216637 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.199613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.27.109.62:37259)
0812 11:15:13.200321 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.200341 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.200391 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:15:13.201479 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.201503 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.202000 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.202423 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.203462 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.204551 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.205458 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.205999 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.206203 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.210031 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.210037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.210057 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:13.210094 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.212347 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.213180 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.213186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.213192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.213447 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.213879 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.213885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.213890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.214471 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.214647 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.215294 (+ 647us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.215703 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":246,"threads_started":1}
I20260812 11:15:13.216671 1895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393: No bootstrap required, opened a new log
I20260812 11:15:13.222723 1895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709247b898a249438c9da96230b1a393" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37259 } }
I20260812 11:15:13.223276 1895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:13.223371 1895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709247b898a249438c9da96230b1a393, State: Initialized, Role: FOLLOWER
I20260812 11:15:13.224056 1895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709247b898a249438c9da96230b1a393" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37259 } }
I20260812 11:15:13.224283 1895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:13.224366 1895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:13.224493 1895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:13.225919 1895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709247b898a249438c9da96230b1a393" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37259 } }
I20260812 11:15:13.226449 1895 leader_election.cc:302] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 709247b898a249438c9da96230b1a393; no voters:
I20260812 11:15:13.227046 1895 leader_election.cc:288] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:13.227365 1903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:13.227730 1903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [term 1 LEADER]: Becoming Leader. State: Replica: 709247b898a249438c9da96230b1a393, State: Running, Role: LEADER
I20260812 11:15:13.228147 1903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709247b898a249438c9da96230b1a393" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37259 } }
I20260812 11:15:13.228855 1895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:13.230451 1906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709247b898a249438c9da96230b1a393. Latest consensus state: current_term: 1 leader_uuid: "709247b898a249438c9da96230b1a393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709247b898a249438c9da96230b1a393" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37259 } } }
I20260812 11:15:13.230518 1905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709247b898a249438c9da96230b1a393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709247b898a249438c9da96230b1a393" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37259 } } }
I20260812 11:15:13.230644 1906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:13.230655 1905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:13.231942 1913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:13.235217 1913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:13.243029 1913 catalog_manager.cc:1347] Generated new cluster ID: fdb644124daf4902a4106728ece241a0
I20260812 11:15:13.243142 1913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:13.250082 1913 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:13.251405 1913 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:13.267681 1913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393: Generated new TSK 0
I20260812 11:15:13.268675 1913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:15:13.322985 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37259
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:13.464175 1930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:13.464483 1930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:13.464535 1930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:13.468962 1930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:13.469049 1930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:13.469133 1930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:13.474905 1930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:13.476233 1930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:13.477393 1930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:13.488755 1930 server_base.cc:1034] running on GCE node
W20260812 11:15:13.488766 1941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.488996 1943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.488765 1940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.489837 1930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:13.490782 1930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:13.492058 1930 hybrid_clock.cc:648] HybridClock initialized: now 1786533313492027 us; error 52 us; skew 500 ppm
Aug 12 11:15:13 dist-test-slave-7z32 krb5kdc[1773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:13.496310 1930 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:15:13.497539 1930 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:13.497684 1930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:13.497993 1930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:13.501037 1930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "44156cbc42c843abb80f57ba437ff98a"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "b576d61985775a596a886c370922f58e"
server_key_iv: "ee91c566535c374c092324126ce4ee93"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.501888 1930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "44156cbc42c843abb80f57ba437ff98a"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "b576d61985775a596a886c370922f58e"
server_key_iv: "ee91c566535c374c092324126ce4ee93"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.508420 1930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 11:15:13.512593 1949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.514719 1930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:13.515049 1930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "44156cbc42c843abb80f57ba437ff98a"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "b576d61985775a596a886c370922f58e"
server_key_iv: "ee91c566535c374c092324126ce4ee93"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.515271 1930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:13.543519 1930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:13.548353 1930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:13.548655 1930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:13.549530 1930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:13.551044 1930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:13.551136 1930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.551236 1930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:13.551282 1930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.564453 1930 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:40785
I20260812 11:15:13.564476 2113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:40785 every 8 connection(s)
I20260812 11:15:13.565420 1930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:13.575485 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 1930
I20260812 11:15:13.575676 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:13.576063 28084 external_mini_cluster.cc:1428] Setting key 9f5cfc33af5d707340a2461d2308dfa4
Aug 12 11:15:13 dist-test-slave-7z32 krb5kdc[1773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:15:13.590304 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.567917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:56635 (local address 127.27.109.62:37259)
0812 11:15:13.568147 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.568154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.569305 (+ 1151us) server_negotiation.cc:407] Connection header received
0812 11:15:13.571017 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.571020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.571088 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.571199 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.573322 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.574048 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.576566 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.576944 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.577077 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.584820 (+ 7743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.584828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.584832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:13.584893 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.587053 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.588037 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.588045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.588049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.588169 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.588643 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.588648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.588651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.588980 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.589117 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.590036 (+ 919us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.590167 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:15:13.591759 2117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.568324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37259 (local address 127.27.109.1:56635)
0812 11:15:13.568997 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.569085 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.570701 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.571633 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.571645 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.572233 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:13.573052 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.573075 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.574299 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.574303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.576284 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.576306 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.577189 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.577193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.577490 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.582051 (+ 4561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:13.582087 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:13.584403 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:13.587303 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:13.587313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:13.587340 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:13.587858 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:13.588292 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:13.588295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:13.588296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:13.588492 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:13.589171 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:13.589180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:13.589698 (+ 518us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.589942 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.590424 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":169,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:13.593828 2115 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37259
I20260812 11:15:13.594260 2115 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:13.595079 2115 heartbeater.cc:511] Master 127.27.109.62:37259 requested a full tablet report, sending...
I20260812 11:15:13.597957 1839 ts_manager.cc:194] Registered new tserver with Master: 44156cbc42c843abb80f57ba437ff98a (127.27.109.1:40785)
I20260812 11:15:13.600224 1839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:56635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:15:13.618650 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37259
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:13.774470 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:13.775283 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:13.775403 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:13.780529 2120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:13.780711 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:13.780835 2120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:13.788097 2120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:13.789947 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:13.791553 2120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:13.805285 2120 server_base.cc:1034] running on GCE node
W20260812 11:15:13.805224 2129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.805253 2127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.805327 2131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.806275 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:13.807374 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:13.808609 2120 hybrid_clock.cc:648] HybridClock initialized: now 1786533313808596 us; error 57 us; skew 500 ppm
Aug 12 11:15:13 dist-test-slave-7z32 krb5kdc[1773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:13.812196 2120 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:15:13.813237 2120 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:13.813314 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:13.813537 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:13.816316 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "e7968cfb50c34f849a88049e6764a500"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "3b5956f1a49ef28f185d2eab041a8337"
server_key_iv: "84ee1a09ed20751cf17cabf3635b8f3c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.817160 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "e7968cfb50c34f849a88049e6764a500"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "3b5956f1a49ef28f185d2eab041a8337"
server_key_iv: "84ee1a09ed20751cf17cabf3635b8f3c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.824028 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260812 11:15:13.827996 2143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.830348 2120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:15:13.830550 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "e7968cfb50c34f849a88049e6764a500"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-7z32"
server_key: "3b5956f1a49ef28f185d2eab041a8337"
server_key_iv: "84ee1a09ed20751cf17cabf3635b8f3c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.830708 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:13.866030 2120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:13.870605 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:13.870944 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:13.871665 2120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:13.872720 2120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:13.872798 2120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.872882 2120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:13.872931 2120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.885510 2120 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45731
I20260812 11:15:13.885532 2315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45731 every 8 connection(s)
I20260812 11:15:13.887145 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:13.890981 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 2120
I20260812 11:15:13.891500 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:13.892340 28084 external_mini_cluster.cc:1428] Setting key 11737cdb8eb4d8a5327704812e30a91d
Aug 12 11:15:13 dist-test-slave-7z32 krb5kdc[1773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:15:13.916352 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.890705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34401 (local address 127.27.109.62:37259)
0812 11:15:13.893437 (+ 2732us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.893444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.896055 (+ 2611us) server_negotiation.cc:407] Connection header received
0812 11:15:13.899022 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.899025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.899232 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.899369 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.902500 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.903346 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.906015 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.906337 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.906630 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.910630 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.910636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.910639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:13.910688 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.912518 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.913443 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.913447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.913448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.913550 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.914197 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.914201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.914202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.914570 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.914727 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.915831 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.916049 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2020}
I20260812 11:15:13.917749 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.894513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37259 (local address 127.27.109.2:34401)
0812 11:15:13.895807 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.895922 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.898422 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.900197 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.900222 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.901166 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:13.902110 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.902137 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.903601 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.903606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.905697 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.905728 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.906918 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.906925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.907206 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.908231 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:13.908258 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:13.910273 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:13.912735 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:13.912743 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:13.912765 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:13.913138 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:13.913745 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:13.913749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:13.913751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:13.913996 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:13.914946 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:13.914961 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:13.915418 (+ 457us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.915707 (+ 289us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.916194 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":317,"threads_started":1}
I20260812 11:15:13.919782 2316 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37259
I20260812 11:15:13.920174 2316 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:13.920958 2316 heartbeater.cc:511] Master 127.27.109.62:37259 requested a full tablet report, sending...
I20260812 11:15:13.922914 1839 ts_manager.cc:194] Registered new tserver with Master: e7968cfb50c34f849a88049e6764a500 (127.27.109.2:45731)
I20260812 11:15:13.923739 1839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:34401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:15:13.943265 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37259
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:14.085527 2323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:14.085912 2323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:14.086004 2323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:14.090435 2323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:14.090566 2323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:14.090673 2323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:14.096913 2323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:14.098273 2323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:14.099565 2323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:14.111663 2330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:14.111747 2323 server_base.cc:1034] running on GCE node
W20260812 11:15:14.111627 2331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:14.111634 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:14.112732 2323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:14.113725 2323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:14.115062 2323 hybrid_clock.cc:648] HybridClock initialized: now 1786533314115031 us; error 52 us; skew 500 ppm
Aug 12 11:15:14 dist-test-slave-7z32 krb5kdc[1773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:14.119342 2323 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:15:14.120476 2323 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:14.120564 2323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:14.120828 2323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:14.123490 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "cd46cfb420be47a08d0fae5c771e8ebd"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-7z32"
server_key: "8e77553551e41df57565bce9acdd5983"
server_key_iv: "11c01845f081e19d1682d2dec04dd947"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.124087 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "cd46cfb420be47a08d0fae5c771e8ebd"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-7z32"
server_key: "8e77553551e41df57565bce9acdd5983"
server_key_iv: "11c01845f081e19d1682d2dec04dd947"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.128876 2323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:15:14.132575 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:14.134636 2323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:14.134927 2323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "cd46cfb420be47a08d0fae5c771e8ebd"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-7z32"
server_key: "8e77553551e41df57565bce9acdd5983"
server_key_iv: "11c01845f081e19d1682d2dec04dd947"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.135069 2323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:14.154382 2323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:14.158694 2323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:14.158980 2323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:14.159608 2323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:14.160621 2323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:14.160683 2323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:14.160727 2323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:14.160743 2323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:14.172978 2323 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:41385
I20260812 11:15:14.173207 2517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:41385 every 8 connection(s)
I20260812 11:15:14.173769 2323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:14.174424 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 2323
I20260812 11:15:14.174594 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:14.175251 28084 external_mini_cluster.cc:1428] Setting key a45d7f1f7bce37df5f4f96c386f773a9
Aug 12 11:15:14 dist-test-slave-7z32 krb5kdc[1773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:14.200402 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.176904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:42475 (local address 127.27.109.62:37259)
0812 11:15:14.177120 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.177126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.179366 (+ 2240us) server_negotiation.cc:407] Connection header received
0812 11:15:14.182184 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.182189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.182289 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.182421 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:14.185194 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.186002 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.188228 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.188509 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.188822 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.193600 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:14.193605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:14.193608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:14.193650 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:14.196068 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:14.197326 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:14.197331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:14.197334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:14.197487 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:14.198199 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:14.198203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:14.198206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:14.198531 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:14.198662 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.200006 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.200207 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 11:15:14.201864 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.177593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37259 (local address 127.27.109.3:42475)
0812 11:15:14.179022 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.179135 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.181446 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.182739 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.182751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.183842 (+ 1091us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:14.184851 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.184881 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.186355 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.186360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.187980 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.188004 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.188915 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.188933 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.189394 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.190383 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:14.190412 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:14.193296 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:14.196360 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:14.196372 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:14.196398 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:14.196983 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:14.197655 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:14.197658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:14.197660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:14.197983 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:14.198707 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:14.198716 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:14.199526 (+ 810us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.199841 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.200334 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":467,"threads_started":1}
I20260812 11:15:14.203920 2518 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37259
I20260812 11:15:14.204351 2518 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:14.205202 2518 heartbeater.cc:511] Master 127.27.109.62:37259 requested a full tablet report, sending...
I20260812 11:15:14.206625 1839 ts_manager.cc:194] Registered new tserver with Master: cd46cfb420be47a08d0fae5c771e8ebd (127.27.109.3:41385)
I20260812 11:15:14.207561 1839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:42475
I20260812 11:15:14.218596 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:14.233943 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.221992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.27.109.62:37259)
0812 11:15:14.222250 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.222255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.222529 (+ 274us) server_negotiation.cc:407] Connection header received
0812 11:15:14.222657 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.222660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.222729 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.223127 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:14.224246 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.224878 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.225965 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.226298 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.226414 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.228454 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:14.228459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:14.228461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:14.228685 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:14.231093 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:14.232010 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:14.232016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:14.232018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:14.232150 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:14.232641 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:14.232647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:14.232652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:14.233024 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:14.233174 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.233576 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.233776 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260812 11:15:14.238904 1839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:14.242589 1839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:14.264689 2541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.256789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.27.109.2:45731)
0812 11:15:14.258771 (+ 1982us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.258783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.258807 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:15:14.258938 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.259020 (+ 82us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.259230 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.259391 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.259827 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.261127 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.263052 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.263888 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.264002 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.264167 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.264382 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.264503 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1833,"thread_start_us":198,"threads_started":1}
I20260812 11:15:14.265518 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.257018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.27.109.1:40785)
0812 11:15:14.257551 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.257559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.258440 (+ 881us) server_negotiation.cc:407] Connection header received
0812 11:15:14.258566 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.258573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.258964 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.259173 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.259561 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.260510 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.262541 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.263602 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.263752 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.263998 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.265228 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.265353 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":162,"threads_started":1}
I20260812 11:15:14.265517 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.257869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.27.109.3:41385)
0812 11:15:14.258579 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.258591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.258619 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:15:14.258713 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.258719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.259122 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.259703 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.259898 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.261155 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.262807 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.263659 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.264899 (+ 1240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.265115 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.265190 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.265324 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":127,"threads_started":1}
I20260812 11:15:14.264691 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.256637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:40785 (local address 127.0.0.1:33984)
0812 11:15:14.257835 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.257912 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.258210 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.259163 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.259171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.259201 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.259413 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.259421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.260647 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.260655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.262332 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.262346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.263785 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.263790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.263943 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.264129 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.264219 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.264426 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":92,"spinlock_wait_cycles":10368,"thread_start_us":110,"threads_started":1}
I20260812 11:15:14.264726 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.257620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41385 (local address 127.0.0.1:38374)
0812 11:15:14.258317 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.258334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.258433 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.259336 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.259340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.259360 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.259537 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.259548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.261320 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.261324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.262608 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.262620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.263860 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.263864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.263947 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.264062 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.264191 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.264414 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":197,"thread_start_us":217,"threads_started":1}
I20260812 11:15:14.265435 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.256654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45731 (local address 127.0.0.1:57004)
0812 11:15:14.257831 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.257906 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.258197 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.259582 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.259588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.259608 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.259696 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.259701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.261320 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.261324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.262894 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.262906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.264046 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.264049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.264208 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.264233 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.264317 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.265290 (+ 973us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":132,"spinlock_wait_cycles":22656,"thread_start_us":213,"threads_started":1}
I20260812 11:15:14.270354 2017 tablet_service.cc:1467] Processing CreateTablet for tablet 10a429af12a64ac5b354a2be40d55eb0 (DEFAULT_TABLE table=test-table [id=c9b52152f4bd4eeaa800172dc7d18021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:14.270429 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 10a429af12a64ac5b354a2be40d55eb0 (DEFAULT_TABLE table=test-table [id=c9b52152f4bd4eeaa800172dc7d18021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:14.271777 2017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a429af12a64ac5b354a2be40d55eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:14.271941 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a429af12a64ac5b354a2be40d55eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:14.273484 2215 tablet_service.cc:1467] Processing CreateTablet for tablet 10a429af12a64ac5b354a2be40d55eb0 (DEFAULT_TABLE table=test-table [id=c9b52152f4bd4eeaa800172dc7d18021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:14.274920 2215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a429af12a64ac5b354a2be40d55eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:14.282591 2546 tablet_bootstrap.cc:492] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Bootstrap starting.
I20260812 11:15:14.283401 2547 tablet_bootstrap.cc:492] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a: Bootstrap starting.
I20260812 11:15:14.284204 2551 tablet_bootstrap.cc:492] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Bootstrap starting.
I20260812 11:15:14.285368 2546 tablet_bootstrap.cc:654] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:14.286671 2546 log.cc:824] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:14.286703 2547 tablet_bootstrap.cc:654] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:14.286698 2551 tablet_bootstrap.cc:654] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:14.287885 2547 log.cc:824] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:14.288174 2551 log.cc:824] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:14.289181 2546 tablet_bootstrap.cc:492] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: No bootstrap required, opened a new log
I20260812 11:15:14.289440 2546 ts_tablet_manager.cc:1397] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:15:14.290459 2547 tablet_bootstrap.cc:492] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a: No bootstrap required, opened a new log
I20260812 11:15:14.290660 2551 tablet_bootstrap.cc:492] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: No bootstrap required, opened a new log
I20260812 11:15:14.290710 2547 ts_tablet_manager.cc:1397] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:15:14.290930 2551 ts_tablet_manager.cc:1397] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 11:15:14.293735 2546 raft_consensus.cc:348] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.294080 2546 raft_consensus.cc:374] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:14.294155 2546 raft_consensus.cc:729] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd46cfb420be47a08d0fae5c771e8ebd, State: Initialized, Role: FOLLOWER
I20260812 11:15:14.294691 2546 consensus_queue.cc:260] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.294888 2551 raft_consensus.cc:348] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.295161 2551 raft_consensus.cc:374] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:14.295215 2551 raft_consensus.cc:729] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7968cfb50c34f849a88049e6764a500, State: Initialized, Role: FOLLOWER
I20260812 11:15:14.295043 2547 raft_consensus.cc:348] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.295408 2547 raft_consensus.cc:374] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:14.295460 2547 raft_consensus.cc:729] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44156cbc42c843abb80f57ba437ff98a, State: Initialized, Role: FOLLOWER
I20260812 11:15:14.295629 2551 consensus_queue.cc:260] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.296033 2546 ts_tablet_manager.cc:1428] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260812 11:15:14.296535 2518 heartbeater.cc:503] Master 127.27.109.62:37259 was elected leader, sending a full tablet report...
I20260812 11:15:14.296666 2551 ts_tablet_manager.cc:1428] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:14.296695 2316 heartbeater.cc:503] Master 127.27.109.62:37259 was elected leader, sending a full tablet report...
I20260812 11:15:14.297242 2547 consensus_queue.cc:260] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.298202 2115 heartbeater.cc:503] Master 127.27.109.62:37259 was elected leader, sending a full tablet report...
I20260812 11:15:14.298693 2547 ts_tablet_manager.cc:1428] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260812 11:15:14.317180 2116 tablet.cc:2369] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:15:14.388927 2317 tablet.cc:2369] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:14.420712 2563 raft_consensus.cc:482] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:14.421028 2563 raft_consensus.cc:504] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.422606 2563 leader_election.cc:288] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7968cfb50c34f849a88049e6764a500 (127.27.109.2:45731), cd46cfb420be47a08d0fae5c771e8ebd (127.27.109.3:41385)
W20260812 11:15:14.425139 2523 tablet.cc:2369] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:14.428748 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.423459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41385 (local address 127.27.109.1:50059)
0812 11:15:14.423957 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.423974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.424110 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.424461 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.424464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.424485 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.424686 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.424695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.425816 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.425820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.426976 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.426984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.427987 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.427991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.428212 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.428281 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.428392 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.428522 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":113,"thread_start_us":144,"threads_started":1}
I20260812 11:15:14.428828 2541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.423101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:54599 (local address 127.27.109.2:45731)
0812 11:15:14.423370 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.423377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.423830 (+ 453us) server_negotiation.cc:407] Connection header received
0812 11:15:14.423912 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.423915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.423998 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.424098 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.424588 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.425247 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.427056 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.427780 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.427957 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.428028 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.428502 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.428681 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 11:15:14.428897 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.423731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50059 (local address 127.27.109.3:41385)
0812 11:15:14.423918 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.423923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.424136 (+ 213us) server_negotiation.cc:407] Connection header received
0812 11:15:14.424211 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.424214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.424273 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.424385 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.424916 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.425673 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.427093 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.427780 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.427957 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.428028 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.428498 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.428635 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 11:15:14.428748 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.422905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45731 (local address 127.27.109.1:54599)
0812 11:15:14.423427 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.423451 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.423650 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.424176 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.424182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.424206 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.424431 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.424439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.425429 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.425434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.426787 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.426800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.427987 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.427991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.428216 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.428281 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.428391 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.428522 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":75,"thread_start_us":170,"threads_started":1}
I20260812 11:15:14.430436 2245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a429af12a64ac5b354a2be40d55eb0" candidate_uuid: "44156cbc42c843abb80f57ba437ff98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7968cfb50c34f849a88049e6764a500" is_pre_election: true
I20260812 11:15:14.430436 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a429af12a64ac5b354a2be40d55eb0" candidate_uuid: "44156cbc42c843abb80f57ba437ff98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" is_pre_election: true
I20260812 11:15:14.431037 2245 raft_consensus.cc:2454] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44156cbc42c843abb80f57ba437ff98a in term 0.
I20260812 11:15:14.431037 2447 raft_consensus.cc:2454] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44156cbc42c843abb80f57ba437ff98a in term 0.
I20260812 11:15:14.431805 1957 leader_election.cc:302] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44156cbc42c843abb80f57ba437ff98a, e7968cfb50c34f849a88049e6764a500; no voters:
I20260812 11:15:14.432225 2563 raft_consensus.cc:2781] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:14.432335 2563 raft_consensus.cc:482] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:14.432384 2563 raft_consensus.cc:3037] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:14.433624 2563 raft_consensus.cc:504] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.434062 2563 leader_election.cc:288] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [CANDIDATE]: Term 1 election: Requested vote from peers e7968cfb50c34f849a88049e6764a500 (127.27.109.2:45731), cd46cfb420be47a08d0fae5c771e8ebd (127.27.109.3:41385)
I20260812 11:15:14.434701 2245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a429af12a64ac5b354a2be40d55eb0" candidate_uuid: "44156cbc42c843abb80f57ba437ff98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7968cfb50c34f849a88049e6764a500"
I20260812 11:15:14.434875 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a429af12a64ac5b354a2be40d55eb0" candidate_uuid: "44156cbc42c843abb80f57ba437ff98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd46cfb420be47a08d0fae5c771e8ebd"
I20260812 11:15:14.434969 2245 raft_consensus.cc:3037] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:14.435128 2447 raft_consensus.cc:3037] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:14.436421 2245 raft_consensus.cc:2454] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44156cbc42c843abb80f57ba437ff98a in term 1.
I20260812 11:15:14.436971 1957 leader_election.cc:302] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44156cbc42c843abb80f57ba437ff98a, e7968cfb50c34f849a88049e6764a500; no voters:
I20260812 11:15:14.437004 2447 raft_consensus.cc:2454] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44156cbc42c843abb80f57ba437ff98a in term 1.
I20260812 11:15:14.437223 2563 raft_consensus.cc:2781] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:14.437551 2563 raft_consensus.cc:686] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [term 1 LEADER]: Becoming Leader. State: Replica: 44156cbc42c843abb80f57ba437ff98a, State: Running, Role: LEADER
I20260812 11:15:14.437997 2563 consensus_queue.cc:237] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } }
I20260812 11:15:14.443836 1839 catalog_manager.cc:5495] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a reported cstate change: term changed from 0 to 1, leader changed from <none> to 44156cbc42c843abb80f57ba437ff98a (127.27.109.1). New cstate: current_term: 1 leader_uuid: "44156cbc42c843abb80f57ba437ff98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44156cbc42c843abb80f57ba437ff98a" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:14.472805 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.467788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.27.109.1:40785)
0812 11:15:14.467986 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.467994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.468123 (+ 129us) server_negotiation.cc:407] Connection header received
0812 11:15:14.468384 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.468388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.468587 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.468736 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:14.469067 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.469658 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.470671 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.471136 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.471420 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.471821 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:14.472328 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:14.472478 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.472547 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.472619 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 11:15:14.483320 2447 raft_consensus.cc:1261] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 1 FOLLOWER]: Refusing update from remote peer 44156cbc42c843abb80f57ba437ff98a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:14.483384 2245 raft_consensus.cc:1261] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 1 FOLLOWER]: Refusing update from remote peer 44156cbc42c843abb80f57ba437ff98a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:14.484659 2567 consensus_queue.cc:1035] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd46cfb420be47a08d0fae5c771e8ebd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:14.485109 2563 consensus_queue.cc:1035] T 10a429af12a64ac5b354a2be40d55eb0 P 44156cbc42c843abb80f57ba437ff98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7968cfb50c34f849a88049e6764a500" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:14.498980 2572 mvcc.cc:205] Tried to move back new op lower bound from 7317640456107704320 to 7317640455941906432. Current Snapshot: MvccSnapshot[applied={T|T < 7317640456107704320}]
I20260812 11:15:14.500703 2571 mvcc.cc:205] Tried to move back new op lower bound from 7317640456107704320 to 7317640455941906432. Current Snapshot: MvccSnapshot[applied={T|T < 7317640456107704320}]
I20260812 11:15:14.510563 1836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:14.511144 1836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:14.515846 1836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 709247b898a249438c9da96230b1a393: Sending DeleteTablet for 3 replicas of tablet 10a429af12a64ac5b354a2be40d55eb0
I20260812 11:15:14.517446 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a429af12a64ac5b354a2be40d55eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:14 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:38374
I20260812 11:15:14.517534 2017 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a429af12a64ac5b354a2be40d55eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:14 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:33984
I20260812 11:15:14.517794 2215 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a429af12a64ac5b354a2be40d55eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:14 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57004
I20260812 11:15:14.518528 2583 tablet_replica.cc:330] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: stopping tablet replica
I20260812 11:15:14.519052 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 1930
I20260812 11:15:14.519079 2583 raft_consensus.cc:2229] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:14.519497 2583 raft_consensus.cc:2258] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:14.529654 2581 tablet_replica.cc:330] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: stopping tablet replica
I20260812 11:15:14.530243 2581 raft_consensus.cc:2229] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:14.530750 2581 raft_consensus.cc:2258] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:14.533015 2581 ts_tablet_manager.cc:1905] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:14.533008 2583 ts_tablet_manager.cc:1905] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:14.536844 2583 ts_tablet_manager.cc:1918] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:14.537045 2583 log.cc:1196] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/10a429af12a64ac5b354a2be40d55eb0
I20260812 11:15:14.537441 2583 ts_tablet_manager.cc:1939] T 10a429af12a64ac5b354a2be40d55eb0 P e7968cfb50c34f849a88049e6764a500: Deleting consensus metadata
I20260812 11:15:14.537405 2581 ts_tablet_manager.cc:1918] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:14.537559 2581 log.cc:1196] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/10a429af12a64ac5b354a2be40d55eb0
I20260812 11:15:14.537990 2581 ts_tablet_manager.cc:1939] T 10a429af12a64ac5b354a2be40d55eb0 P cd46cfb420be47a08d0fae5c771e8ebd: Deleting consensus metadata
I20260812 11:15:14.539556 1820 catalog_manager.cc:4841] TS e7968cfb50c34f849a88049e6764a500 (127.27.109.2:45731): tablet 10a429af12a64ac5b354a2be40d55eb0 (table test-table [id=c9b52152f4bd4eeaa800172dc7d18021]) successfully deleted
I20260812 11:15:14.539944 1820 catalog_manager.cc:4841] TS cd46cfb420be47a08d0fae5c771e8ebd (127.27.109.3:41385): tablet 10a429af12a64ac5b354a2be40d55eb0 (table test-table [id=c9b52152f4bd4eeaa800172dc7d18021]) successfully deleted
W20260812 11:15:14.540221 1814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:40785 (error 108)
I20260812 11:15:14.540565 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 2120
I20260812 11:15:14.544248 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.543822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:40785 (local address 127.0.0.1:34000)
0812 11:15:14.544025 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.544125 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 11:15:14.544678 1814 catalog_manager.cc:4584] TS 44156cbc42c843abb80f57ba437ff98a (127.27.109.1:40785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10a429af12a64ac5b354a2be40d55eb0: Network error: Client connection negotiation failed: client connection to 127.27.109.1:40785: connect: Connection refused (error 111)
I20260812 11:15:14.552462 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 2323
I20260812 11:15:14.617895 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.617314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:40785 (local address 127.0.0.1:34014)
0812 11:15:14.617630 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.617757 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 11:15:14.618968 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.618702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:40785 (local address 127.0.0.1:34016)
0812 11:15:14.618813 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.618896 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 11:15:14.624395 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 1791
2026-08-12T11:15:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3166 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T11:15:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:14Z Disabled control of system clock
I20260812 11:15:14.678454 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33245
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 11:15:14.825753 2590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:14.826098 2590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:14.826141 2590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:14.832453 2590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:14.832593 2590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:14.832643 2590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:14.832669 2590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:14.832698 2590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:14.841426 2590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:14.843678 2590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:14.846179 2590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:14.859484 2599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:14.859489 2600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:14.860977 2603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:14.862957 2590 server_base.cc:1034] running on GCE node
I20260812 11:15:14.864048 2590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:14.865943 2590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:14.869319 2590 hybrid_clock.cc:648] HybridClock initialized: now 1786533314869190 us; error 137 us; skew 500 ppm
I20260812 11:15:14.872931 2590 webserver.cc:460] Webserver started at http://127.27.109.62:40519/ using document root <none> and password file <none>
I20260812 11:15:14.873899 2590 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:14.874037 2590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:14.874383 2590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:14.878039 2590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-7z32"
server_key: "1036cb21346bb32cf3c76991469f7d4e"
server_key_iv: "2ef2553467b5c4aac1d747240eb1d0ca"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.879078 2590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-7z32"
server_key: "1036cb21346bb32cf3c76991469f7d4e"
server_key_iv: "2ef2553467b5c4aac1d747240eb1d0ca"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.884673 2590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 11:15:14.888585 2614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:14.890689 2590 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:15:14.891029 2590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-7z32"
server_key: "1036cb21346bb32cf3c76991469f7d4e"
server_key_iv: "2ef2553467b5c4aac1d747240eb1d0ca"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.891494 2590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:14.898257 2590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:14.899459 2590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:14.899735 2590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:14.908950 2590 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:33245
I20260812 11:15:14.908998 2687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:33245 every 8 connection(s)
I20260812 11:15:14.910218 2590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:14.912748 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 2590
I20260812 11:15:14.912923 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:14.913311 28084 external_mini_cluster.cc:1428] Setting key 3a1ce10b1e419906d9ed43bb6cb55764
I20260812 11:15:14.916502 2688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:14.921598 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.915649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.27.109.62:33245)
0812 11:15:14.916896 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.916917 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.916975 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:15:14.917763 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.917786 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.918271 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.918644 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:14.919921 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:14.919927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:14.919943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:14.919957 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:14.920029 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:14.920048 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:14.920227 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.920349 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.920689 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":237,"threads_started":1}
I20260812 11:15:14.925422 2688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8: Bootstrap starting.
I20260812 11:15:14.928797 2688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:14.929916 2688 log.cc:824] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:14.933045 2688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8: No bootstrap required, opened a new log
I20260812 11:15:14.938367 2688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33245 } }
I20260812 11:15:14.939011 2688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:14.939173 2688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ccaf6ca41479c9c7c03a10e5689d8, State: Initialized, Role: FOLLOWER
I20260812 11:15:14.939924 2688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33245 } }
I20260812 11:15:14.940169 2688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:14.940243 2688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:14.940347 2688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:14.941744 2688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33245 } }
I20260812 11:15:14.942237 2688 leader_election.cc:302] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b7ccaf6ca41479c9c7c03a10e5689d8; no voters:
I20260812 11:15:14.942683 2688 leader_election.cc:288] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:14.943035 2698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:14.943488 2698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7ccaf6ca41479c9c7c03a10e5689d8, State: Running, Role: LEADER
I20260812 11:15:14.943948 2698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33245 } }
I20260812 11:15:14.944621 2688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:14.946151 2701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7ccaf6ca41479c9c7c03a10e5689d8. Latest consensus state: current_term: 1 leader_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33245 } } }
I20260812 11:15:14.946341 2701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:14.946723 2700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ccaf6ca41479c9c7c03a10e5689d8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33245 } } }
I20260812 11:15:14.946862 2700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:14.947010 2709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:14.951153 2709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:14.959636 2709 catalog_manager.cc:1347] Generated new cluster ID: ac6a75ce0c0949a18a8e23f0fe100d91
I20260812 11:15:14.959761 2709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:14.972038 2709 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:14.972503 2709 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:14.982263 2709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8: Generated new TSK 0
I20260812 11:15:14.983377 2709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:14.993973 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33245
--builtin_ntp_servers=127.27.109.20:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 11:15:15.161310 2723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:15.161760 2723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:15.161811 2723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:15.167140 2723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:15.167248 2723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:15.167320 2723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:15.173310 2723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:15.175199 2723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:15.177356 2723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:15.193320 2730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:15.193305 2733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:15.193503 2731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:15.193934 2723 server_base.cc:1034] running on GCE node
I20260812 11:15:15.194620 2723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:15.195787 2723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:15.197077 2723 hybrid_clock.cc:648] HybridClock initialized: now 1786533315197045 us; error 52 us; skew 500 ppm
I20260812 11:15:15.200101 2723 webserver.cc:460] Webserver started at http://127.27.109.1:45999/ using document root <none> and password file <none>
I20260812 11:15:15.201009 2723 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:15.201128 2723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:15.201435 2723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:15.204622 2723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "c7c311b76c804303b706d77877a704a3"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "9f98171d6f09eb411b24888717b60861"
server_key_iv: "b670d6f9d31ca8ef04123194c8833b8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.205416 2723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "c7c311b76c804303b706d77877a704a3"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "9f98171d6f09eb411b24888717b60861"
server_key_iv: "b670d6f9d31ca8ef04123194c8833b8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.210939 2723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 11:15:15.215167 2743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.217145 2723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:15:15.217386 2723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "c7c311b76c804303b706d77877a704a3"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "9f98171d6f09eb411b24888717b60861"
server_key_iv: "b670d6f9d31ca8ef04123194c8833b8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.217604 2723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:15.224656 2723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:15.225673 2723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:15.225950 2723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:15.226779 2723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:15.228209 2723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:15.228294 2723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.228380 2723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:15.228427 2723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.242672 2723 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:45195
I20260812 11:15:15.242729 2915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:45195 every 8 connection(s)
I20260812 11:15:15.244067 2723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:15.248569 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 2723
I20260812 11:15:15.248701 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:15.249073 28084 external_mini_cluster.cc:1428] Setting key b5b23d374523c16b310ea2ad3d9c224b
I20260812 11:15:15.252272 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33245
--builtin_ntp_servers=127.27.109.20:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 11:15:15.253561 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.246627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:42689 (local address 127.27.109.62:33245)
0812 11:15:15.246960 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.246965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.248434 (+ 1469us) server_negotiation.cc:407] Connection header received
0812 11:15:15.250608 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.250612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.250661 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.250755 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.252808 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.252815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.252821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.252841 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.252876 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.252898 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.253029 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.253301 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.253364 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 11:15:15.255306 2921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.247167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33245 (local address 127.27.109.1:42689)
0812 11:15:15.248203 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.248265 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.250270 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.251239 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.251253 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.251423 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.252278 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.252281 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.252418 (+ 137us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.252572 (+ 154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.253074 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.253084 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.253090 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.253351 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.253839 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":182,"threads_started":1}
I20260812 11:15:15.257277 2916 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33245
I20260812 11:15:15.257699 2916 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:15.258432 2916 heartbeater.cc:511] Master 127.27.109.62:33245 requested a full tablet report, sending...
I20260812 11:15:15.260830 2632 ts_manager.cc:194] Registered new tserver with Master: c7c311b76c804303b706d77877a704a3 (127.27.109.1:45195)
W20260812 11:15:15.387741 2923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:15.388127 2923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:15.388172 2923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:15.393646 2923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:15.393775 2923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:15.393877 2923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:15.399518 2923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:15.400950 2923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:15.402578 2923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:15.415535 2923 server_base.cc:1034] running on GCE node
W20260812 11:15:15.415413 2929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:15.415467 2931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:15.415467 2933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:15.416481 2923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:15.417294 2923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:15.418540 2923 hybrid_clock.cc:648] HybridClock initialized: now 1786533315418521 us; error 82 us; skew 500 ppm
I20260812 11:15:15.421042 2923 webserver.cc:460] Webserver started at http://127.27.109.2:43263/ using document root <none> and password file <none>
I20260812 11:15:15.421747 2923 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:15.421895 2923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:15.422199 2923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:15.424748 2923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "39a606b36abb49459f8fa8f77c4a0395"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "00feba0f1fa144f5ed50fdfd2636bd64"
server_key_iv: "381aea6c04fa384702f7b7d47a64e4ca"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.425396 2923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "39a606b36abb49459f8fa8f77c4a0395"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "00feba0f1fa144f5ed50fdfd2636bd64"
server_key_iv: "381aea6c04fa384702f7b7d47a64e4ca"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.430133 2923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:15:15.433696 2946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.435434 2923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:15.435627 2923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "39a606b36abb49459f8fa8f77c4a0395"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "00feba0f1fa144f5ed50fdfd2636bd64"
server_key_iv: "381aea6c04fa384702f7b7d47a64e4ca"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.435775 2923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:15.441167 2923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:15.441891 2923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:15.442104 2923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:15.442723 2923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:15.443809 2923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:15.443897 2923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.443989 2923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:15.444033 2923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.455724 2923 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:40817
I20260812 11:15:15.455869 3109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:40817 every 8 connection(s)
I20260812 11:15:15.456969 2923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:15.463130 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 2923
I20260812 11:15:15.463253 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:15.463585 28084 external_mini_cluster.cc:1428] Setting key 2ad49025358b6edfc77ad7d70c1c974e
I20260812 11:15:15.465752 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.459809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59773 (local address 127.27.109.62:33245)
0812 11:15:15.460052 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.460059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.461572 (+ 1513us) server_negotiation.cc:407] Connection header received
0812 11:15:15.463443 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.463447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.463514 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.463633 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.465137 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.465141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.465143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.465160 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.465193 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.465203 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.465348 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.465553 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.465609 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 11:15:15.466306 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33245
--builtin_ntp_servers=127.27.109.20:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 11:15:15.466984 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.460236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33245 (local address 127.27.109.2:59773)
0812 11:15:15.461333 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.461398 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.463089 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.463865 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.463873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.463991 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.464745 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.464747 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.464814 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.464922 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.465349 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.465368 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.465379 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.465612 (+ 233us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.465925 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":145,"threads_started":1}
I20260812 11:15:15.468482 3112 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33245
I20260812 11:15:15.468782 3112 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:15.469395 3112 heartbeater.cc:511] Master 127.27.109.62:33245 requested a full tablet report, sending...
I20260812 11:15:15.471328 2632 ts_manager.cc:194] Registered new tserver with Master: 39a606b36abb49459f8fa8f77c4a0395 (127.27.109.2:40817)
W20260812 11:15:15.625406 3117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:15.625852 3117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:15.625905 3117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:15.630625 3117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:15.630728 3117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:15.630797 3117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:15.637081 3117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:15.638718 3117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:15.640933 3117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:15.655377 3123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:15.655578 3126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:15.655426 3124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:15.656170 3117 server_base.cc:1034] running on GCE node
I20260812 11:15:15.656941 3117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:15.657919 3117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:15.661283 3117 hybrid_clock.cc:648] HybridClock initialized: now 1786533315661242 us; error 118 us; skew 500 ppm
I20260812 11:15:15.664587 3117 webserver.cc:460] Webserver started at http://127.27.109.3:33045/ using document root <none> and password file <none>
I20260812 11:15:15.665488 3117 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:15.665614 3117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:15.665939 3117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:15.668926 3117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "23720088d8d849eeb28f03679ab66691"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "432db364f1daa15b1783c094a1fdb578"
server_key_iv: "0e7113ce52a2a1365a0158e04c0b6377"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.669793 3117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "23720088d8d849eeb28f03679ab66691"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "432db364f1daa15b1783c094a1fdb578"
server_key_iv: "0e7113ce52a2a1365a0158e04c0b6377"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.676061 3117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:15:15.680858 3137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.683421 3117 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260812 11:15:15.683658 3117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "23720088d8d849eeb28f03679ab66691"
format_stamp: "Formatted at 2026-08-12 11:15:15 on dist-test-slave-7z32"
server_key: "432db364f1daa15b1783c094a1fdb578"
server_key_iv: "0e7113ce52a2a1365a0158e04c0b6377"
server_key_version: "encryptionkey@0"
I20260812 11:15:15.684036 3117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:15.690064 3117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:15.690904 3117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:15.691130 3117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:15.691954 3117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:15.693266 3117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:15.693363 3117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.693449 3117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:15.693495 3117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:15.706274 3117 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:40561
I20260812 11:15:15.706292 3297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:40561 every 8 connection(s)
I20260812 11:15:15.707571 3117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:15.710515 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 3117
I20260812 11:15:15.710676 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:15.711161 28084 external_mini_cluster.cc:1428] Setting key 6907994edbf08b713da9eabe8bd79f52
I20260812 11:15:15.716190 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.710421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:40763 (local address 127.27.109.62:33245)
0812 11:15:15.710665 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.710672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.712431 (+ 1759us) server_negotiation.cc:407] Connection header received
0812 11:15:15.714016 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.714021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.714084 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.714195 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.715606 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.715609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.715612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.715625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.715653 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.715661 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.715750 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.715991 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.716041 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 11:15:15.717415 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.711063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33245 (local address 127.27.109.3:40763)
0812 11:15:15.712243 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.712310 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.713778 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.714377 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.714385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.714507 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.715294 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.715296 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.715346 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.715432 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.715765 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.715774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.715780 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.715981 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.716247 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":232,"threads_started":1}
I20260812 11:15:15.719336 3298 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33245
I20260812 11:15:15.719712 3298 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:15.720382 3298 heartbeater.cc:511] Master 127.27.109.62:33245 requested a full tablet report, sending...
I20260812 11:15:15.722093 2632 ts_manager.cc:194] Registered new tserver with Master: 23720088d8d849eeb28f03679ab66691 (127.27.109.3:40561)
I20260812 11:15:15.726421 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:15.731069 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.728839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.27.109.62:33245)
0812 11:15:15.728984 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.728987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.729294 (+ 307us) server_negotiation.cc:407] Connection header received
0812 11:15:15.729474 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.729477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.729516 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.729611 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.730488 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.730491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.730493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.730505 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.730527 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.730534 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.730621 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.730816 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.730953 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:15:15.735543 2632 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:15.740000 2632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:15.757047 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.752839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:45195 (local address 127.0.0.1:47914)
0812 11:15:15.753896 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.753916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.754052 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.754934 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.754943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.755037 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.755813 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.755815 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.755841 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.755923 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.756547 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.756553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.756559 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.756643 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.756761 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":146,"threads_started":1}
I20260812 11:15:15.757068 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.753524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.27.109.2:40817)
0812 11:15:15.754105 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.754119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.754382 (+ 263us) server_negotiation.cc:407] Connection header received
0812 11:15:15.754603 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.754610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.754903 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.755042 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.756238 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.756244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.756255 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.756266 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.756309 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.756332 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.756478 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.756714 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.756843 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":203,"threads_started":1}
I20260812 11:15:15.757251 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.752736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.27.109.3:40561)
0812 11:15:15.753574 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.753591 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.753639 (+ 48us) server_negotiation.cc:407] Connection header received
0812 11:15:15.753856 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.753865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.754173 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.754475 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.756371 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.756378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.756386 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.756401 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.756447 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.756482 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.756588 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.756982 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.757069 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":239,"threads_started":1}
I20260812 11:15:15.757652 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.752398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:40561 (local address 127.0.0.1:38050)
0812 11:15:15.753493 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.753551 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.753675 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.755483 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.755488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.755513 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.756177 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.756180 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.756192 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.756217 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.756641 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.756645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.756649 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.756736 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.756805 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":5,"thread_start_us":131,"threads_started":1}
I20260812 11:15:15.757839 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.753267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:40817 (local address 127.0.0.1:47578)
0812 11:15:15.754269 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.754284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.754461 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.755432 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.755437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.755458 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.755926 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.755928 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.755947 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.755988 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.756447 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.756457 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.756464 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.756643 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.756790 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":601,"thread_start_us":99,"threads_started":1}
I20260812 11:15:15.758526 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.753154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.27.109.1:45195)
0812 11:15:15.753622 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.753629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.754173 (+ 544us) server_negotiation.cc:407] Connection header received
0812 11:15:15.754360 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.754371 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.754577 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.754726 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.756246 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.756257 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.756280 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.756295 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.756356 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.756389 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.756542 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.758205 (+ 1663us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.758320 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":168,"threads_started":1}
I20260812 11:15:15.761590 3018 tablet_service.cc:1467] Processing CreateTablet for tablet 8822664bb3f34a0eb00fb6a43228dfd2 (DEFAULT_TABLE table=test-table [id=684bd7ffb17c450f9980f9f38391b023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:15.761577 3206 tablet_service.cc:1467] Processing CreateTablet for tablet 8822664bb3f34a0eb00fb6a43228dfd2 (DEFAULT_TABLE table=test-table [id=684bd7ffb17c450f9980f9f38391b023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:15.761758 2812 tablet_service.cc:1467] Processing CreateTablet for tablet 8822664bb3f34a0eb00fb6a43228dfd2 (DEFAULT_TABLE table=test-table [id=684bd7ffb17c450f9980f9f38391b023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:15.763038 3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8822664bb3f34a0eb00fb6a43228dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:15.763187 2812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8822664bb3f34a0eb00fb6a43228dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:15.763829 3018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8822664bb3f34a0eb00fb6a43228dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:15.770591 3324 tablet_bootstrap.cc:492] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Bootstrap starting.
I20260812 11:15:15.773535 3324 tablet_bootstrap.cc:654] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:15.774122 3325 tablet_bootstrap.cc:492] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3: Bootstrap starting.
I20260812 11:15:15.774899 3324 log.cc:824] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:15.776211 3327 tablet_bootstrap.cc:492] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Bootstrap starting.
I20260812 11:15:15.777341 3325 tablet_bootstrap.cc:654] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:15.778528 3325 log.cc:824] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:15.779019 3324 tablet_bootstrap.cc:492] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: No bootstrap required, opened a new log
I20260812 11:15:15.779330 3324 ts_tablet_manager.cc:1397] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 11:15:15.779488 3327 tablet_bootstrap.cc:654] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:15.781013 3327 log.cc:824] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:15.783035 3325 tablet_bootstrap.cc:492] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3: No bootstrap required, opened a new log
I20260812 11:15:15.783340 3325 ts_tablet_manager.cc:1397] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 11:15:15.784004 3327 tablet_bootstrap.cc:492] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: No bootstrap required, opened a new log
I20260812 11:15:15.784240 3327 ts_tablet_manager.cc:1397] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:15:15.785326 3324 raft_consensus.cc:348] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.785758 3324 raft_consensus.cc:374] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:15.785821 3324 raft_consensus.cc:729] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23720088d8d849eeb28f03679ab66691, State: Initialized, Role: FOLLOWER
I20260812 11:15:15.786484 3324 consensus_queue.cc:260] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.788105 3324 ts_tablet_manager.cc:1428] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20260812 11:15:15.788647 3298 heartbeater.cc:503] Master 127.27.109.62:33245 was elected leader, sending a full tablet report...
I20260812 11:15:15.788997 3327 raft_consensus.cc:348] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.788866 3325 raft_consensus.cc:348] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.789320 3327 raft_consensus.cc:374] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:15.789320 3325 raft_consensus.cc:374] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:15.789393 3327 raft_consensus.cc:729] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a606b36abb49459f8fa8f77c4a0395, State: Initialized, Role: FOLLOWER
I20260812 11:15:15.789393 3325 raft_consensus.cc:729] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c311b76c804303b706d77877a704a3, State: Initialized, Role: FOLLOWER
I20260812 11:15:15.789953 3327 consensus_queue.cc:260] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.789953 3325 consensus_queue.cc:260] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.791263 3325 ts_tablet_manager.cc:1428] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260812 11:15:15.791440 3112 heartbeater.cc:503] Master 127.27.109.62:33245 was elected leader, sending a full tablet report...
I20260812 11:15:15.791801 2916 heartbeater.cc:503] Master 127.27.109.62:33245 was elected leader, sending a full tablet report...
I20260812 11:15:15.792553 3327 ts_tablet_manager.cc:1428] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 11:15:15.907904 3333 raft_consensus.cc:482] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:15.908259 3333 raft_consensus.cc:504] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.909840 3333 leader_election.cc:288] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23720088d8d849eeb28f03679ab66691 (127.27.109.3:40561), c7c311b76c804303b706d77877a704a3 (127.27.109.1:45195)
I20260812 11:15:15.912659 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.910925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34685 (local address 127.27.109.3:40561)
0812 11:15:15.911134 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.911139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.911154 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:15:15.911211 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.911216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.911255 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.911368 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.912218 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.912221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.912223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.912235 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.912257 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.912265 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.912438 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.912474 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.912515 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 11:15:15.912689 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.910283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:40561 (local address 127.27.109.2:34685)
0812 11:15:15.910764 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.910780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.910897 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.911429 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.911435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.911451 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.911962 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.911964 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.911979 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.912014 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.912359 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.912363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.912367 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.912446 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.912497 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":179,"threads_started":1}
I20260812 11:15:15.912976 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.910371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:44165 (local address 127.27.109.1:45195)
0812 11:15:15.910508 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.910514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.910734 (+ 220us) server_negotiation.cc:407] Connection header received
0812 11:15:15.910815 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.910851 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.910908 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.910995 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.912233 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.912238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.912242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.912261 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.912299 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.912312 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.912433 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.912743 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.912820 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:15:15.913332 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.910224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:45195 (local address 127.27.109.2:44165)
0812 11:15:15.910485 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:15.910503 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:15.910637 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:15.911118 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:15.911121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:15.911136 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:15.911962 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:15.911964 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:15.911980 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:15.912014 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:15.912488 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:15.912493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:15.912497 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 11:15:15.912578 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 11:15:15.912631 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260812 11:15:15.913880 3230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822664bb3f34a0eb00fb6a43228dfd2" candidate_uuid: "39a606b36abb49459f8fa8f77c4a0395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23720088d8d849eeb28f03679ab66691" is_pre_election: true
I20260812 11:15:15.914113 2838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822664bb3f34a0eb00fb6a43228dfd2" candidate_uuid: "39a606b36abb49459f8fa8f77c4a0395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c311b76c804303b706d77877a704a3" is_pre_election: true
I20260812 11:15:15.914291 3230 raft_consensus.cc:2454] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a606b36abb49459f8fa8f77c4a0395 in term 0.
I20260812 11:15:15.914496 2838 raft_consensus.cc:2454] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a606b36abb49459f8fa8f77c4a0395 in term 0.
I20260812 11:15:15.914938 2949 leader_election.cc:302] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23720088d8d849eeb28f03679ab66691, 39a606b36abb49459f8fa8f77c4a0395; no voters:
I20260812 11:15:15.915320 3333 raft_consensus.cc:2781] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:15.915431 3333 raft_consensus.cc:482] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:15.915481 3333 raft_consensus.cc:3037] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:15.916800 3333 raft_consensus.cc:504] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.917292 3333 leader_election.cc:288] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [CANDIDATE]: Term 1 election: Requested vote from peers 23720088d8d849eeb28f03679ab66691 (127.27.109.3:40561), c7c311b76c804303b706d77877a704a3 (127.27.109.1:45195)
I20260812 11:15:15.917783 3230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822664bb3f34a0eb00fb6a43228dfd2" candidate_uuid: "39a606b36abb49459f8fa8f77c4a0395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23720088d8d849eeb28f03679ab66691"
I20260812 11:15:15.917906 2838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822664bb3f34a0eb00fb6a43228dfd2" candidate_uuid: "39a606b36abb49459f8fa8f77c4a0395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c311b76c804303b706d77877a704a3"
I20260812 11:15:15.918009 3230 raft_consensus.cc:3037] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:15.918108 2838 raft_consensus.cc:3037] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:15.919548 2838 raft_consensus.cc:2454] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a606b36abb49459f8fa8f77c4a0395 in term 1.
I20260812 11:15:15.919780 3230 raft_consensus.cc:2454] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a606b36abb49459f8fa8f77c4a0395 in term 1.
I20260812 11:15:15.919991 2950 leader_election.cc:302] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a606b36abb49459f8fa8f77c4a0395, c7c311b76c804303b706d77877a704a3; no voters:
I20260812 11:15:15.920234 3333 raft_consensus.cc:2781] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:15.920559 3333 raft_consensus.cc:686] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 1 LEADER]: Becoming Leader. State: Replica: 39a606b36abb49459f8fa8f77c4a0395, State: Running, Role: LEADER
I20260812 11:15:15.921037 3333 consensus_queue.cc:237] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } }
I20260812 11:15:15.926247 2633 catalog_manager.cc:5495] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a606b36abb49459f8fa8f77c4a0395 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "39a606b36abb49459f8fa8f77c4a0395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a606b36abb49459f8fa8f77c4a0395" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 } health_report { overall_health: UNKNOWN } } }
W20260812 11:15:15.958485 3114 tablet.cc:2369] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:15:15.959008 3299 tablet.cc:2369] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:15.963167 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:15.960808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.27.109.2:40817)
0812 11:15:15.961003 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:15.961011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:15.961081 (+ 70us) server_negotiation.cc:407] Connection header received
0812 11:15:15.961392 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:15.961395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:15.961471 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:15.961587 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:15.962579 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:15.962584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:15.962593 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:15.962606 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:15.962626 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:15.962634 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:15.962744 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:15.962934 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 11:15:15.963014 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 11:15:15.971489 2838 raft_consensus.cc:1261] T 8822664bb3f34a0eb00fb6a43228dfd2 P c7c311b76c804303b706d77877a704a3 [term 1 FOLLOWER]: Refusing update from remote peer 39a606b36abb49459f8fa8f77c4a0395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:15.971546 3230 raft_consensus.cc:1261] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 1 FOLLOWER]: Refusing update from remote peer 39a606b36abb49459f8fa8f77c4a0395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:15.972754 3336 consensus_queue.cc:1035] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23720088d8d849eeb28f03679ab66691" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:15.973253 3333 consensus_queue.cc:1035] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c311b76c804303b706d77877a704a3" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:15.982960 3342 mvcc.cc:205] Tried to move back new op lower bound from 7317640462207152128 to 7317640462016368640. Current Snapshot: MvccSnapshot[applied={T|T < 7317640462207152128}]
I20260812 11:15:15.986213 3344 mvcc.cc:205] Tried to move back new op lower bound from 7317640462207152128 to 7317640462016368640. Current Snapshot: MvccSnapshot[applied={T|T < 7317640462207152128}]
I20260812 11:15:15.998000 2638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:15.998225 2638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:16.002246 2638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b7ccaf6ca41479c9c7c03a10e5689d8: Sending DeleteTablet for 3 replicas of tablet 8822664bb3f34a0eb00fb6a43228dfd2
I20260812 11:15:16.003492 3206 tablet_service.cc:1514] Processing DeleteTablet for tablet 8822664bb3f34a0eb00fb6a43228dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:15 UTC) from {username='slave'} at 127.0.0.1:38050
I20260812 11:15:16.003716 2812 tablet_service.cc:1514] Processing DeleteTablet for tablet 8822664bb3f34a0eb00fb6a43228dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:15 UTC) from {username='slave'} at 127.0.0.1:47914
I20260812 11:15:16.003724 3018 tablet_service.cc:1514] Processing DeleteTablet for tablet 8822664bb3f34a0eb00fb6a43228dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:15 UTC) from {username='slave'} at 127.0.0.1:47578
I20260812 11:15:16.004201 3360 tablet_replica.cc:330] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: stopping tablet replica
I20260812 11:15:16.004441 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 2723
I20260812 11:15:16.004673 3360 raft_consensus.cc:2229] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:16.005115 3360 raft_consensus.cc:2258] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:16.021409 3363 tablet_replica.cc:330] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: stopping tablet replica
I20260812 11:15:16.022002 3363 raft_consensus.cc:2229] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:16.022648 3363 raft_consensus.cc:2258] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:16.024659 3360 ts_tablet_manager.cc:1905] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:16.024755 3363 ts_tablet_manager.cc:1905] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:16.028650 3363 ts_tablet_manager.cc:1918] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:16.028650 3360 ts_tablet_manager.cc:1918] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:16.028836 3363 log.cc:1196] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/8822664bb3f34a0eb00fb6a43228dfd2
I20260812 11:15:16.028836 3360 log.cc:1196] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/8822664bb3f34a0eb00fb6a43228dfd2
I20260812 11:15:16.029251 3363 ts_tablet_manager.cc:1939] T 8822664bb3f34a0eb00fb6a43228dfd2 P 39a606b36abb49459f8fa8f77c4a0395: Deleting consensus metadata
I20260812 11:15:16.029258 3360 ts_tablet_manager.cc:1939] T 8822664bb3f34a0eb00fb6a43228dfd2 P 23720088d8d849eeb28f03679ab66691: Deleting consensus metadata
I20260812 11:15:16.031499 2617 catalog_manager.cc:4841] TS 39a606b36abb49459f8fa8f77c4a0395 (127.27.109.2:40817): tablet 8822664bb3f34a0eb00fb6a43228dfd2 (table test-table [id=684bd7ffb17c450f9980f9f38391b023]) successfully deleted
I20260812 11:15:16.031858 2617 catalog_manager.cc:4841] TS 23720088d8d849eeb28f03679ab66691 (127.27.109.3:40561): tablet 8822664bb3f34a0eb00fb6a43228dfd2 (table test-table [id=684bd7ffb17c450f9980f9f38391b023]) successfully deleted
W20260812 11:15:16.032146 2619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:45195 (error 108)
I20260812 11:15:16.033411 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 2923
I20260812 11:15:16.033851 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.033384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:45195 (local address 127.0.0.1:47930)
0812 11:15:16.033600 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.033705 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260812 11:15:16.057049 2619 catalog_manager.cc:4584] TS c7c311b76c804303b706d77877a704a3 (127.27.109.1:45195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8822664bb3f34a0eb00fb6a43228dfd2: Network error: Client connection negotiation failed: client connection to 127.27.109.1:45195: connect: Connection refused (error 111)
I20260812 11:15:16.077046 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 3117
I20260812 11:15:16.098551 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 2590
2026-08-12T11:15:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1472 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 11:15:16.136854 28084 mini_oidc.cc:152] Starting JWKS server
I20260812 11:15:16.139343 28084 webserver.cc:460] Webserver started at https://127.0.0.1:33803/ using document root <none> and password file <none>
I20260812 11:15:16.139634 28084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:15:16.140017 28084 webserver.cc:460] Webserver started at http://127.0.0.1:45275/ using document root <none> and password file <none>
2026-08-12T11:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:16Z Disabled control of system clock
I20260812 11:15:16.155157 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:40641
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:40641
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33803/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/testchainca.cert with env {}
W20260812 11:15:16.294586 3375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:16.294953 3375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:16.294999 3375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:16.299304 3375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:16.299400 3375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:16.299417 3375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:16.299435 3375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:16.299450 3375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:15:16.299467 3375 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:15:16.299481 3375 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33803/jwks.json
I20260812 11:15:16.305604 3375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:40641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33803/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:16.306890 3375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:16.308015 3375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:16.318863 3381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.318912 3384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.318912 3386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:16.319766 3375 server_base.cc:1034] running on GCE node
I20260812 11:15:16.320531 3375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:16.321929 3375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:16.325352 3375 hybrid_clock.cc:648] HybridClock initialized: now 1786533316325229 us; error 80 us; skew 500 ppm
I20260812 11:15:16.328011 3375 webserver.cc:460] Webserver started at http://127.27.109.62:45899/ using document root <none> and password file <none>
I20260812 11:15:16.328715 3375 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:16.328794 3375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:16.328991 3375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:16.331427 3375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "bf64386952a24b0c99e6e4cdb5927bf6"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-7z32"
server_key: "0da60f608bd354be6841fcdfff356b24"
server_key_iv: "aca685133b0c14c16ddf4a9018018651"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.332024 3375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "bf64386952a24b0c99e6e4cdb5927bf6"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-7z32"
server_key: "0da60f608bd354be6841fcdfff356b24"
server_key_iv: "aca685133b0c14c16ddf4a9018018651"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.336947 3375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:15:16.340597 3396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.342563 3375 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:16.342725 3375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "bf64386952a24b0c99e6e4cdb5927bf6"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-7z32"
server_key: "0da60f608bd354be6841fcdfff356b24"
server_key_iv: "aca685133b0c14c16ddf4a9018018651"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.342928 3375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:16.358537 3375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:16.359450 3375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:16.359639 3375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:16.368443 3375 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:40641
I20260812 11:15:16.368501 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:40641 every 8 connection(s)
I20260812 11:15:16.369630 3375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:16.374404 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:16.375618 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 3375
I20260812 11:15:16.375802 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:16.376122 28084 external_mini_cluster.cc:1428] Setting key 278c254aa1f97e94426bd6f5d51f410e
I20260812 11:15:16.383970 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6: Bootstrap starting.
I20260812 11:15:16.388214 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:16.389580 3480 log.cc:824] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:16.390318 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.378126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.27.109.62:40641)
0812 11:15:16.378980 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.378999 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.379039 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:15:16.379928 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.379948 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.380437 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.380788 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:16.381508 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.382636 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.384281 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.385356 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.385615 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.386974 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:16.386982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:16.387006 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:16.387699 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:16.387808 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:16.387846 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:16.388004 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.388262 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.388937 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":195,"threads_started":1}
I20260812 11:15:16.393256 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6: No bootstrap required, opened a new log
I20260812 11:15:16.395383 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:16.400252 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40641 } }
I20260812 11:15:16.400667 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:16.400759 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf64386952a24b0c99e6e4cdb5927bf6, State: Initialized, Role: FOLLOWER
I20260812 11:15:16.401407 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40641 } }
I20260812 11:15:16.401579 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:16.401690 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:16.401851 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:16.403445 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40641 } }
I20260812 11:15:16.403972 3480 leader_election.cc:302] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf64386952a24b0c99e6e4cdb5927bf6; no voters:
I20260812 11:15:16.404495 3480 leader_election.cc:288] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:16.404791 3486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:16.405243 3486 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [term 1 LEADER]: Becoming Leader. State: Replica: bf64386952a24b0c99e6e4cdb5927bf6, State: Running, Role: LEADER
I20260812 11:15:16.405719 3486 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40641 } }
I20260812 11:15:16.406421 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:16.407948 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf64386952a24b0c99e6e4cdb5927bf6. Latest consensus state: current_term: 1 leader_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40641 } } }
I20260812 11:15:16.408130 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:16.408696 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:16.410672 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf64386952a24b0c99e6e4cdb5927bf6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40641 } } }
I20260812 11:15:16.410816 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:16.413826 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:16.422374 3501 catalog_manager.cc:1347] Generated new cluster ID: 616a7f94aafe46219bfb5bcb01cca27f
I20260812 11:15:16.422478 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:16.440131 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:16.441201 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:16.447688 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf64386952a24b0c99e6e4cdb5927bf6: Generated new TSK 0
I20260812 11:15:16.448762 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 11:15:16.497386 3532 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 11:15:16.497649 3532 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 11:15:16.498216 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.468000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.27.109.62:40641)
0812 11:15:16.468161 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.468166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.468447 (+ 281us) server_negotiation.cc:407] Connection header received
0812 11:15:16.468748 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.468751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.468810 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.468924 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:15:16.469385 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.469972 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.470983 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.471225 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.471381 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.471805 (+ 424us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:15:16.497262 (+ 25457us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33803/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 11:15:16.498017 (+ 755us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33803/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 11:15:16.498353 3483 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33803/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 11:15:16.499979 3530 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:16.467844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:40641 (local address 127.0.0.1:54610)
0812 11:15:16.468327 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.468343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.468535 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.468958 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.468961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.468969 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:15:16.469164 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.469171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.470106 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.470110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.470755 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.470763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.471394 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.471398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.471506 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.471572 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:15:16.497990 (+ 26418us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33803/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 11:15:16.498410 (+ 420us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:40641: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33803/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":171,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.109.62:40641: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33803/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 11:15:16.501181 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 3375
2026-08-12T11:15:16Z chronyd exiting
I20260812 11:15:16.565703 28084 test_util.cc:175] -----------------------------------------------
I20260812 11:15:16.565825 28084 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533282802791-28084-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (431 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 11:15:16.567147 28084 mini_oidc.cc:152] Starting JWKS server
I20260812 11:15:16.568135 28084 webserver.cc:460] Webserver started at https://127.0.0.1:37015/ using document root <none> and password file <none>
I20260812 11:15:16.568229 28084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:15:16.568526 28084 webserver.cc:460] Webserver started at http://127.0.0.1:39711/ using document root <none> and password file <none>
2026-08-12T11:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:16Z Disabled control of system clock
I20260812 11:15:16.582341 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36085
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36085
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37015/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/testchainca.cert with env {}
W20260812 11:15:16.715489 3544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:16.715857 3544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:16.715906 3544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:16.720713 3544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:16.720816 3544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:16.720834 3544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:16.720850 3544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:16.720867 3544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:15:16.720885 3544 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:15:16.720899 3544 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37015/jwks.json
I20260812 11:15:16.727166 3544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37015/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:16.728509 3544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:16.729763 3544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:16.740384 3558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.740356 3553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.740343 3556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:16.740808 3544 server_base.cc:1034] running on GCE node
I20260812 11:15:16.741768 3544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:16.743544 3544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:16.744834 3544 hybrid_clock.cc:648] HybridClock initialized: now 1786533316744768 us; error 85 us; skew 500 ppm
I20260812 11:15:16.747922 3544 webserver.cc:460] Webserver started at http://127.27.109.62:38777/ using document root <none> and password file <none>
I20260812 11:15:16.748832 3544 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:16.748962 3544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:16.749289 3544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:16.753165 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "e52dddee8f79455aa2a677b3c0632222"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-7z32"
server_key: "7f17e463b8848d7b8a7d47df07ad086b"
server_key_iv: "b1768192d02654ad576ee3f65f2de935"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.754323 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "e52dddee8f79455aa2a677b3c0632222"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-7z32"
server_key: "7f17e463b8848d7b8a7d47df07ad086b"
server_key_iv: "b1768192d02654ad576ee3f65f2de935"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.761155 3544 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 11:15:16.765580 3567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.767890 3544 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:15:16.768181 3544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "e52dddee8f79455aa2a677b3c0632222"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-7z32"
server_key: "7f17e463b8848d7b8a7d47df07ad086b"
server_key_iv: "b1768192d02654ad576ee3f65f2de935"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.768392 3544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:16.782626 3544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:16.783924 3544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:16.784257 3544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:16.794407 3544 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:36085
I20260812 11:15:16.794495 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:36085 every 8 connection(s)
I20260812 11:15:16.795854 3544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:16.800128 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:16.804000 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 3544
I20260812 11:15:16.804157 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:16.804483 28084 external_mini_cluster.cc:1428] Setting key 553dce4992aea751a0576df52d872241
I20260812 11:15:16.807668 3648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222: Bootstrap starting.
I20260812 11:15:16.811668 3648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:16.813014 3648 log.cc:824] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:16.817816 3648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222: No bootstrap required, opened a new log
I20260812 11:15:16.824541 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.806735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.27.109.62:36085)
0812 11:15:16.808457 (+ 1722us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.808507 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.808572 (+ 65us) server_negotiation.cc:407] Connection header received
0812 11:15:16.809956 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.809998 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.811159 (+ 1161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.812152 (+ 993us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:16.814143 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.816497 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.818389 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.819407 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.819646 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.821386 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:16.821395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:16.821426 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:16.822232 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:16.822354 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:16.822401 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:16.822559 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.823003 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.823450 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":215,"threads_started":1}
I20260812 11:15:16.824528 3648 raft_consensus.cc:348] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52dddee8f79455aa2a677b3c0632222" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36085 } }
I20260812 11:15:16.825055 3648 raft_consensus.cc:374] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:16.825132 3648 raft_consensus.cc:729] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52dddee8f79455aa2a677b3c0632222, State: Initialized, Role: FOLLOWER
I20260812 11:15:16.825754 3648 consensus_queue.cc:260] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52dddee8f79455aa2a677b3c0632222" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36085 } }
I20260812 11:15:16.825937 3648 raft_consensus.cc:388] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:16.826009 3648 raft_consensus.cc:482] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:16.826121 3648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:16.828002 3648 raft_consensus.cc:504] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52dddee8f79455aa2a677b3c0632222" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36085 } }
I20260812 11:15:16.828483 3648 leader_election.cc:302] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e52dddee8f79455aa2a677b3c0632222; no voters:
I20260812 11:15:16.828989 3648 leader_election.cc:288] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:16.829169 3658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:16.829463 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:16.829538 3658 raft_consensus.cc:686] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [term 1 LEADER]: Becoming Leader. State: Replica: e52dddee8f79455aa2a677b3c0632222, State: Running, Role: LEADER
I20260812 11:15:16.829957 3658 consensus_queue.cc:237] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52dddee8f79455aa2a677b3c0632222" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36085 } }
I20260812 11:15:16.830940 3648 sys_catalog.cc:564] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:16.831969 3660 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e52dddee8f79455aa2a677b3c0632222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52dddee8f79455aa2a677b3c0632222" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36085 } } }
I20260812 11:15:16.832150 3660 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:16.832544 3661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e52dddee8f79455aa2a677b3c0632222. Latest consensus state: current_term: 1 leader_uuid: "e52dddee8f79455aa2a677b3c0632222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52dddee8f79455aa2a677b3c0632222" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36085 } } }
I20260812 11:15:16.832659 3661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:16.832933 3671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:16.836648 3671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:16.843473 3671 catalog_manager.cc:1347] Generated new cluster ID: 0abe9571df2141238bb8b61c8b4af5c7
I20260812 11:15:16.843609 3671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:16.876756 3671 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:16.878068 3671 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:16.888123 3671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e52dddee8f79455aa2a677b3c0632222: Generated new TSK 0
I20260812 11:15:16.889535 3671 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 11:15:16.929106 3691 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 11:15:16.929253 3691 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 11:15:16.930372 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.910586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.27.109.62:36085)
0812 11:15:16.910798 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.910805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.911009 (+ 204us) server_negotiation.cc:407] Connection header received
0812 11:15:16.911243 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.911248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.911334 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.911542 (+ 208us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:15:16.911834 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.912878 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.914002 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.914444 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.914598 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.915143 (+ 545us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:15:16.929724 (+ 14581us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 11:15:16.930163 (+ 439us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":92}
W20260812 11:15:16.930451 3690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:16.910406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36085 (local address 127.0.0.1:53390)
0812 11:15:16.910805 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.910845 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.911031 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.911545 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.911547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.911555 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:15:16.911695 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.911701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.913059 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.913062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.913845 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.913855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.914649 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.914653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.914751 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.914788 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:15:16.930053 (+ 15265us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 11:15:16.930270 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:36085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":174,"threads_started":1}
W20260812 11:15:16.930497 3655 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260812 11:15:16.931596 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 3544
2026-08-12T11:15:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (435 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 11:15:17.003507 28084 mini_oidc.cc:152] Starting JWKS server
I20260812 11:15:17.004752 28084 webserver.cc:460] Webserver started at https://127.0.0.1:37775/ using document root <none> and password file <none>
I20260812 11:15:17.004940 28084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:15:17.005298 28084 webserver.cc:460] Webserver started at http://127.0.0.1:42183/ using document root <none> and password file <none>
2026-08-12T11:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:17Z Disabled control of system clock
I20260812 11:15:17.021881 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35655
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:35655
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37775/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 11:15:17.152412 3700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:17.152985 3700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:17.153137 3700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:17.157826 3700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:17.157972 3700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:17.158003 3700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:17.158051 3700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:17.158107 3700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:15:17.158149 3700 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:15:17.158181 3700 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37775/jwks.json
I20260812 11:15:17.163648 3700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:35655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37775/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:17.165035 3700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:17.166934 3700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:17.179283 3712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.179345 3714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:17.179572 3700 server_base.cc:1034] running on GCE node
W20260812 11:15:17.179888 3711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:17.180884 3700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:17.182811 3700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:17.184302 3700 hybrid_clock.cc:648] HybridClock initialized: now 1786533317184249 us; error 80 us; skew 500 ppm
I20260812 11:15:17.187482 3700 webserver.cc:460] Webserver started at http://127.27.109.62:43567/ using document root <none> and password file <none>
I20260812 11:15:17.188405 3700 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:17.188545 3700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:17.188897 3700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:17.192068 3700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "7131507bf4104a2dbe8b5dc08434edb3"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-7z32"
server_key: "cee7d78c4ad75e9a8e2852ad844da0ea"
server_key_iv: "0a7514a33530b1a88426084c32489694"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.192988 3700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "7131507bf4104a2dbe8b5dc08434edb3"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-7z32"
server_key: "cee7d78c4ad75e9a8e2852ad844da0ea"
server_key_iv: "0a7514a33530b1a88426084c32489694"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.198575 3700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:15:17.202541 3724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.204722 3700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:17.204923 3700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "7131507bf4104a2dbe8b5dc08434edb3"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-7z32"
server_key: "cee7d78c4ad75e9a8e2852ad844da0ea"
server_key_iv: "0a7514a33530b1a88426084c32489694"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.205106 3700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:17.220227 3700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:17.221201 3700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:17.221410 3700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:17.232385 3700 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:35655
I20260812 11:15:17.232421 3820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:35655 every 8 connection(s)
I20260812 11:15:17.233934 3700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:17.237926 3822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:17.243999 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 3700
I20260812 11:15:17.244217 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:17.244422 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3: Bootstrap starting.
I20260812 11:15:17.244797 28084 external_mini_cluster.cc:1428] Setting key e4cdfda660fd74b0a4027887ae678ac0
I20260812 11:15:17.247474 3822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:17.248454 3822 log.cc:824] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:17.251303 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3: No bootstrap required, opened a new log
I20260812 11:15:17.255851 3822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7131507bf4104a2dbe8b5dc08434edb3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35655 } }
I20260812 11:15:17.256186 3822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:17.256244 3822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7131507bf4104a2dbe8b5dc08434edb3, State: Initialized, Role: FOLLOWER
I20260812 11:15:17.256875 3822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7131507bf4104a2dbe8b5dc08434edb3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35655 } }
I20260812 11:15:17.257067 3822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:17.257167 3822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:17.257316 3822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:17.258270 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.246773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.27.109.62:35655)
0812 11:15:17.247820 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.247841 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.247896 (+ 55us) server_negotiation.cc:407] Connection header received
0812 11:15:17.248851 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.248879 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.249307 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.249668 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:17.250355 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.251904 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.253142 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.253894 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.254076 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.255336 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:17.255342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:17.255361 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:17.255925 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:17.256007 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:17.256036 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:17.256157 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.256526 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.256998 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":202,"threads_started":1}
I20260812 11:15:17.258507 3822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7131507bf4104a2dbe8b5dc08434edb3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35655 } }
I20260812 11:15:17.258976 3822 leader_election.cc:302] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7131507bf4104a2dbe8b5dc08434edb3; no voters:
I20260812 11:15:17.259392 3822 leader_election.cc:288] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:17.259618 3831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:17.260205 3831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [term 1 LEADER]: Becoming Leader. State: Replica: 7131507bf4104a2dbe8b5dc08434edb3, State: Running, Role: LEADER
I20260812 11:15:17.260699 3831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7131507bf4104a2dbe8b5dc08434edb3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35655 } }
I20260812 11:15:17.260864 3822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:17.262082 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:17.263137 3835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7131507bf4104a2dbe8b5dc08434edb3. Latest consensus state: current_term: 1 leader_uuid: "7131507bf4104a2dbe8b5dc08434edb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7131507bf4104a2dbe8b5dc08434edb3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35655 } } }
I20260812 11:15:17.263355 3835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:17.263792 3832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7131507bf4104a2dbe8b5dc08434edb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7131507bf4104a2dbe8b5dc08434edb3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35655 } } }
I20260812 11:15:17.264295 3832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3 [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:17.266703 3856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:17.266808 3856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:17.266942 3858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:17.267774 3858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:17.275915 3858 catalog_manager.cc:1347] Generated new cluster ID: d016b8de5db9466abf1477b4fd84a58e
I20260812 11:15:17.276031 3858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:17.289306 3858 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:17.290472 3858 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:17.295239 3858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7131507bf4104a2dbe8b5dc08434edb3: Generated new TSK 0
I20260812 11:15:17.296025 3858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:17.313835 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.307499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.27.109.62:35655)
0812 11:15:17.307668 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.307672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.307915 (+ 243us) server_negotiation.cc:407] Connection header received
0812 11:15:17.308149 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.308151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.308214 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.308333 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:15:17.308791 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.309374 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.311003 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.311315 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.311431 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.311909 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:15:17.313393 (+ 1484us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37775/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 11:15:17.313662 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37775/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 11:15:17.313941 3829 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37775/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 11:15:17.316155 3872 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:17.307252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35655 (local address 127.0.0.1:41222)
0812 11:15:17.307730 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:17.307753 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:17.307967 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:17.308403 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:17.308410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:17.308425 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:15:17.308637 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.308646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.309588 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.309592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.310862 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.310875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.311490 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.311498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.311608 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.311665 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:15:17.313609 (+ 1944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37775/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 11:15:17.315901 (+ 2292us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:35655: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37775/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":1944,"thread_start_us":158,"threads_started":1}
I20260812 11:15:17.316926 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 3700
E20260812 11:15:17.361351 3874 webserver.cc:533] Webserver: sslize failed:
E20260812 11:15:17.361500 3874 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T11:15:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (391 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T11:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:17Z Disabled control of system clock
I20260812 11:15:17.412014 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46105
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:46105
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/testchainca.cert with env {}
W20260812 11:15:17.568431 3879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:17.568876 3879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:17.568929 3879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:17.574472 3879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:17.574604 3879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:17.574621 3879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:17.574643 3879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:17.574662 3879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:15:17.574683 3879 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:15:17.574699 3879 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 11:15:17.582957 3879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:46105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:17.584834 3879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:17.586352 3879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:17.598624 3887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.598738 3890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:17.598686 3879 server_base.cc:1034] running on GCE node
W20260812 11:15:17.598701 3888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:17.599880 3879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:17.601573 3879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:17.602859 3879 hybrid_clock.cc:648] HybridClock initialized: now 1786533317602783 us; error 74 us; skew 500 ppm
I20260812 11:15:17.605816 3879 webserver.cc:460] Webserver started at http://127.27.109.62:43873/ using document root <none> and password file <none>
I20260812 11:15:17.606637 3879 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:17.606716 3879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:17.607030 3879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:17.609740 3879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "93956041c57b47faa7895e8a6c490e2f"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-7z32"
server_key: "e571619a1c1b6ca95507cd3d9ab180e3"
server_key_iv: "08f933b3df179802a3fa47cc3785c974"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.610486 3879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "93956041c57b47faa7895e8a6c490e2f"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-7z32"
server_key: "e571619a1c1b6ca95507cd3d9ab180e3"
server_key_iv: "08f933b3df179802a3fa47cc3785c974"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.616727 3879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:17.621147 3903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.623629 3879 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:15:17.623889 3879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "93956041c57b47faa7895e8a6c490e2f"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-7z32"
server_key: "e571619a1c1b6ca95507cd3d9ab180e3"
server_key_iv: "08f933b3df179802a3fa47cc3785c974"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.624092 3879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:17.644660 3879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:17.645784 3879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:17.646245 3879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:17.658598 3879 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:46105
I20260812 11:15:17.658650 3989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:46105 every 8 connection(s)
I20260812 11:15:17.660871 3879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:17.664501 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 3879
I20260812 11:15:17.664688 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:17.665128 28084 external_mini_cluster.cc:1428] Setting key cf5b4bb0363146837f2de717b09baac9
I20260812 11:15:17.665767 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:17.675534 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f: Bootstrap starting.
I20260812 11:15:17.679061 3990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:17.679145 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.667328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.27.109.62:46105)
0812 11:15:17.668275 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.668297 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.668352 (+ 55us) server_negotiation.cc:407] Connection header received
0812 11:15:17.669310 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.669341 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.669966 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.670420 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:17.670987 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.672653 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.673893 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.674672 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.674986 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.676395 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:17.676402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:17.676421 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:17.677003 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:17.677093 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:17.677125 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:17.677264 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.677461 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.677779 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":190,"threads_started":1}
I20260812 11:15:17.680088 3990 log.cc:824] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:17.683548 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f: No bootstrap required, opened a new log
I20260812 11:15:17.684294 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:17.688295 3990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93956041c57b47faa7895e8a6c490e2f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46105 } }
I20260812 11:15:17.688597 3990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:17.688635 3990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93956041c57b47faa7895e8a6c490e2f, State: Initialized, Role: FOLLOWER
I20260812 11:15:17.689075 3990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93956041c57b47faa7895e8a6c490e2f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46105 } }
I20260812 11:15:17.689204 3990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:17.689260 3990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:17.689349 3990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:17.690419 3990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93956041c57b47faa7895e8a6c490e2f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46105 } }
I20260812 11:15:17.690781 3990 leader_election.cc:302] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93956041c57b47faa7895e8a6c490e2f; no voters:
I20260812 11:15:17.691242 3990 leader_election.cc:288] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:17.691504 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:17.691888 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [term 1 LEADER]: Becoming Leader. State: Replica: 93956041c57b47faa7895e8a6c490e2f, State: Running, Role: LEADER
I20260812 11:15:17.692421 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93956041c57b47faa7895e8a6c490e2f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46105 } }
I20260812 11:15:17.692998 3990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:17.695013 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93956041c57b47faa7895e8a6c490e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93956041c57b47faa7895e8a6c490e2f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46105 } } }
I20260812 11:15:17.695293 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:17.695693 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93956041c57b47faa7895e8a6c490e2f. Latest consensus state: current_term: 1 leader_uuid: "93956041c57b47faa7895e8a6c490e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93956041c57b47faa7895e8a6c490e2f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46105 } } }
I20260812 11:15:17.696010 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:17.698808 4024 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:17.699004 4024 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:17.699090 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:17.700076 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:17.708288 4025 catalog_manager.cc:1347] Generated new cluster ID: 20152a3c0a9545ddabfc22c02064107b
I20260812 11:15:17.708489 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:17.717032 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:17.718171 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:17.724464 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93956041c57b47faa7895e8a6c490e2f: Generated new TSK 0
I20260812 11:15:17.725296 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:18.216786 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.742743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.27.109.62:46105)
0812 11:15:17.742968 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.742973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.743149 (+ 176us) server_negotiation.cc:407] Connection header received
0812 11:15:17.743377 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.743381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.743466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.743667 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:15:17.743978 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.744911 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.746017 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.746380 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.746504 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.747139 (+ 635us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:15:18.216318 (+469179us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 11:15:18.216636 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50986: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":116}
W20260812 11:15:18.216912 3996 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50986: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 11:15:18.217489 4040 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:17.742573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46105 (local address 127.0.0.1:50986)
0812 11:15:17.743027 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:17.743041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:17.743171 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:17.743671 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:17.743673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:17.743680 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:15:17.743846 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.743852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.745074 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.745077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.745868 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.745879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.746550 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.746553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.746631 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.746666 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:15:18.216798 (+470132us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 11:15:18.217215 (+ 417us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:46105: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":166,"threads_started":1}
I20260812 11:15:18.219084 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 3879
2026-08-12T11:15:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (866 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (184 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (183 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:18 dist-test-slave-7z32 krb5kdc[4062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:18 dist-test-slave-7z32 krb5kdc[4062](info): set up 2 sockets
Aug 12 11:15:18 dist-test-slave-7z32 krb5kdc[4062](info): commencing operation
krb5kdc: starting...
W20260812 11:15:20.119422 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.490s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:20 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:20.304241 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44685
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:44685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:20.459223 4085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:20.459653 4085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:20.459705 4085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:20.464685 4085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:20.464795 4085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:20.464818 4085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:20.464838 4085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:20.464855 4085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:20.471717 4085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:44685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:20.473420 4085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:20.474941 4085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:20.487954 4085 server_base.cc:1034] running on GCE node
W20260812 11:15:20.487854 4091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.487839 4092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.487834 4096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:20.489334 4085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:20.491530 4085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:20.492825 4085 hybrid_clock.cc:648] HybridClock initialized: now 1786533320492781 us; error 76 us; skew 500 ppm
Aug 12 11:15:20 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:20.496951 4085 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:20.498672 4085 webserver.cc:460] Webserver started at http://127.27.109.62:35501/ using document root <none> and password file <none>
I20260812 11:15:20.499650 4085 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:20.499752 4085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:20.500010 4085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:20.503229 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "11aa7f454a3d4098a63ac7286770e3b7"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-7z32"
server_key: "4791f08ec8704387bf97c0332a728543"
server_key_iv: "5e5af040e178afcce06f3fc59eaedade"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.504033 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "11aa7f454a3d4098a63ac7286770e3b7"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-7z32"
server_key: "4791f08ec8704387bf97c0332a728543"
server_key_iv: "5e5af040e178afcce06f3fc59eaedade"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.510182 4085 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:15:20.514256 4114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.516577 4085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:20.516826 4085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "11aa7f454a3d4098a63ac7286770e3b7"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-7z32"
server_key: "4791f08ec8704387bf97c0332a728543"
server_key_iv: "5e5af040e178afcce06f3fc59eaedade"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.517019 4085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:20.539572 4085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:20.544848 4085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:20.545200 4085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:20.558130 4085 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:44685
I20260812 11:15:20.558157 4190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:44685 every 8 connection(s)
I20260812 11:15:20.559947 4085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:20.564265 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:20.567013 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 4085
I20260812 11:15:20.567189 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:20.567569 28084 external_mini_cluster.cc:1428] Setting key 6dbbdaa4e25a69ad95bdea190058af69
I20260812 11:15:20.572760 4191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7: Bootstrap starting.
I20260812 11:15:20.576184 4191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:20.577272 4191 log.cc:824] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:20.580804 4191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7: No bootstrap required, opened a new log
Aug 12 11:15:20 dist-test-slave-7z32 krb5kdc[4062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:20.584468 4191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aa7f454a3d4098a63ac7286770e3b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44685 } }
I20260812 11:15:20.584900 4191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:20.585155 4191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aa7f454a3d4098a63ac7286770e3b7, State: Initialized, Role: FOLLOWER
I20260812 11:15:20.585861 4191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aa7f454a3d4098a63ac7286770e3b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44685 } }
I20260812 11:15:20.586066 4191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:20.586174 4191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:20.586405 4191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:20.588229 4191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aa7f454a3d4098a63ac7286770e3b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44685 } }
I20260812 11:15:20.588896 4191 leader_election.cc:302] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11aa7f454a3d4098a63ac7286770e3b7; no voters:
I20260812 11:15:20.589407 4191 leader_election.cc:288] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:20.589525 4200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:20.589808 4200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [term 1 LEADER]: Becoming Leader. State: Replica: 11aa7f454a3d4098a63ac7286770e3b7, State: Running, Role: LEADER
I20260812 11:15:20.590250 4200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aa7f454a3d4098a63ac7286770e3b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44685 } }
I20260812 11:15:20.591145 4191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:20.592566 4201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11aa7f454a3d4098a63ac7286770e3b7. Latest consensus state: current_term: 1 leader_uuid: "11aa7f454a3d4098a63ac7286770e3b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aa7f454a3d4098a63ac7286770e3b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44685 } } }
I20260812 11:15:20.592756 4201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:20.592777 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.569986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.27.109.62:44685)
0812 11:15:20.570681 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.570703 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.570757 (+ 54us) server_negotiation.cc:407] Connection header received
0812 11:15:20.571783 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.571810 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.572474 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.573237 (+ 763us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:20.574221 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.575615 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.577097 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.578051 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.578351 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.582276 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:20.582285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:20.582311 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:20.582360 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:20.586414 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.587293 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.587302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.587311 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.587520 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.588070 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.588075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.588077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.589276 (+ 1199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:20.589636 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.590688 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.591268 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":232,"threads_started":1}
I20260812 11:15:20.593137 4203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11aa7f454a3d4098a63ac7286770e3b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aa7f454a3d4098a63ac7286770e3b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44685 } } }
I20260812 11:15:20.593291 4203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:20.593724 4216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:20.597478 4216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:20.606226 4216 catalog_manager.cc:1347] Generated new cluster ID: ed3932f90ee54d1db65716bb69588551
I20260812 11:15:20.606355 4216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:20.642253 4216 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:20.643766 4216 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:20.657335 4216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11aa7f454a3d4098a63ac7286770e3b7: Generated new TSK 0
I20260812 11:15:20.658356 4216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:15:20.711436 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44685
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:20.877120 4240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:20.877559 4240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:20.877699 4240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:20.882869 4240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:20.882989 4240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:20.883067 4240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:20.891084 4240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:20.892809 4240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:20.894229 4240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:20.907436 4246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.907426 4247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.907421 4251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:20.907804 4240 server_base.cc:1034] running on GCE node
I20260812 11:15:20.908465 4240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:20.909301 4240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:20.910509 4240 hybrid_clock.cc:648] HybridClock initialized: now 1786533320910487 us; error 53 us; skew 500 ppm
Aug 12 11:15:20 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:20.914548 4240 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:15:20.915892 4240 webserver.cc:460] Webserver started at http://127.27.109.1:33033/ using document root <none> and password file <none>
I20260812 11:15:20.916672 4240 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:20.916781 4240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:20.917074 4240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:20.919673 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "b64ea178e6a14047bae55566dcfc30d8"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-7z32"
server_key: "d7bc4b9ef66abf7e188254d7d9689bc7"
server_key_iv: "11402f907305111902d55adca687e91b"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.920413 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "b64ea178e6a14047bae55566dcfc30d8"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-7z32"
server_key: "d7bc4b9ef66abf7e188254d7d9689bc7"
server_key_iv: "11402f907305111902d55adca687e91b"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.926633 4240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260812 11:15:20.931653 4259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.933804 4240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:20.934064 4240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "b64ea178e6a14047bae55566dcfc30d8"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-7z32"
server_key: "d7bc4b9ef66abf7e188254d7d9689bc7"
server_key_iv: "11402f907305111902d55adca687e91b"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.934247 4240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:20.983784 4240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:20.988557 4240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:20.988862 4240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:20.989676 4240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:20.990964 4240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:20.991206 4240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.991340 4240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:20.991418 4240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.005362 4240 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:36785
I20260812 11:15:21.005429 4436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:36785 every 8 connection(s)
I20260812 11:15:21.006865 4240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:21.015594 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 4240
I20260812 11:15:21.015784 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:21.016185 28084 external_mini_cluster.cc:1428] Setting key fd9661b4dc40955432a87efdf342b1ed
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:21.031498 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.009999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:37709 (local address 127.27.109.62:44685)
0812 11:15:21.010294 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.010532 (+ 238us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.012048 (+ 1516us) server_negotiation.cc:407] Connection header received
0812 11:15:21.014234 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.014238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.014308 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.014429 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.016782 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.017521 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.019491 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.019800 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.019953 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.024325 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.024331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.024334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:21.024380 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.026685 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.028303 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.028308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.028310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.028465 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.029168 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.029172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.029173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.029449 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.029556 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.031155 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.031341 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:15:21.033102 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.010902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44685 (local address 127.27.109.1:37709)
0812 11:15:21.011790 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.011860 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.013525 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.014812 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.014845 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.015460 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:21.016426 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.016453 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.017759 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.017763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.019294 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.019314 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.020059 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.020065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.020325 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.021251 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:21.021276 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:21.023896 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:21.026986 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:21.027000 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:21.027039 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:21.028045 (+ 1006us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:21.028616 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:21.028623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:21.028628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:21.028982 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:21.029603 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:21.029618 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:21.030420 (+ 802us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.030783 (+ 363us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.031406 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":181,"threads_started":1}
I20260812 11:15:21.035427 4437 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44685
I20260812 11:15:21.035951 4437 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:21.037631 4437 heartbeater.cc:511] Master 127.27.109.62:44685 requested a full tablet report, sending...
I20260812 11:15:21.040488 4137 ts_manager.cc:194] Registered new tserver with Master: b64ea178e6a14047bae55566dcfc30d8 (127.27.109.1:36785)
I20260812 11:15:21.042026 4137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:37709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:15:21.060153 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44685
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:21.208451 4445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:21.208860 4445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:21.208920 4445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:21.214030 4445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:21.214136 4445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:21.214203 4445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:21.221632 4445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:21.223194 4445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:21.224511 4445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:21.237367 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.237372 4458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.237372 4460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:21.237969 4445 server_base.cc:1034] running on GCE node
I20260812 11:15:21.238682 4445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:21.239794 4445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:21.241039 4445 hybrid_clock.cc:648] HybridClock initialized: now 1786533321240986 us; error 77 us; skew 500 ppm
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.245522 4445 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:15:21.247025 4445 webserver.cc:460] Webserver started at http://127.27.109.2:37099/ using document root <none> and password file <none>
I20260812 11:15:21.247864 4445 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:21.247952 4445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:21.248255 4445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:21.251356 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "572aab2f1f0c460084a7873cbb3bf4a2"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-7z32"
server_key: "7f47472ee4005efa7ed62c2a3230e91e"
server_key_iv: "b42b990d6a4dfb56ef5f0e7d69e17cf7"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.252202 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "572aab2f1f0c460084a7873cbb3bf4a2"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-7z32"
server_key: "7f47472ee4005efa7ed62c2a3230e91e"
server_key_iv: "b42b990d6a4dfb56ef5f0e7d69e17cf7"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.258056 4445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 11:15:21.262151 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.264384 4445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:21.264631 4445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "572aab2f1f0c460084a7873cbb3bf4a2"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-7z32"
server_key: "7f47472ee4005efa7ed62c2a3230e91e"
server_key_iv: "b42b990d6a4dfb56ef5f0e7d69e17cf7"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.264822 4445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:21.283329 4445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:21.288987 4445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:21.289239 4445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:21.290048 4445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:21.291375 4445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:21.291445 4445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.291498 4445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:21.291514 4445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.305927 4445 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45375
I20260812 11:15:21.306046 4644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45375 every 8 connection(s)
I20260812 11:15:21.307356 4445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:21.312650 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 4445
I20260812 11:15:21.312855 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:21.313302 28084 external_mini_cluster.cc:1428] Setting key 556d6d04ce2a74d054fc0600181ac334
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:21.328083 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.310058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:56837 (local address 127.27.109.62:44685)
0812 11:15:21.310315 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.310321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.311729 (+ 1408us) server_negotiation.cc:407] Connection header received
0812 11:15:21.313472 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.313476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.313551 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.313673 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.315338 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.315895 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.317207 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.317447 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.317561 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.321378 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.321386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.321393 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:21.321444 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.324208 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.325233 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.325240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.325245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.325390 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.326066 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.326070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.326072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.326421 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.326560 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.327652 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.327877 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:15:21.329476 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.310563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44685 (local address 127.27.109.2:56837)
0812 11:15:21.311494 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.311564 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.313188 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.313904 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.313912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.314365 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:21.315169 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.315188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.316035 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.316038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.317062 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.317074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.317630 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.317635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.317812 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.318639 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:21.318665 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:21.321049 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:21.324470 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:21.324482 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:21.324508 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:21.325009 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:21.325630 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:21.325634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:21.325636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:21.325889 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:21.326600 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:21.326610 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:21.327233 (+ 623us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.327497 (+ 264us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.328040 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":181,"threads_started":1}
I20260812 11:15:21.331619 4645 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44685
I20260812 11:15:21.332093 4645 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:15:21.332810 4645 heartbeater.cc:511] Master 127.27.109.62:44685 requested a full tablet report, sending...
I20260812 11:15:21.334528 4137 ts_manager.cc:194] Registered new tserver with Master: 572aab2f1f0c460084a7873cbb3bf4a2 (127.27.109.2:45375)
I20260812 11:15:21.335644 4137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:56837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:15:21.356429 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44685
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:21.515352 4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:21.515777 4651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:21.515836 4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:21.520913 4651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:21.521054 4651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:21.521132 4651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:21.528371 4651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:21.530004 4651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:21.531616 4651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:21.545396 4663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.545418 4661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.545418 4657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:21.545979 4651 server_base.cc:1034] running on GCE node
I20260812 11:15:21.546597 4651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:21.547523 4651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:21.550930 4651 hybrid_clock.cc:648] HybridClock initialized: now 1786533321550855 us; error 73 us; skew 500 ppm
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.554664 4651 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:15:21.555953 4651 webserver.cc:460] Webserver started at http://127.27.109.3:38877/ using document root <none> and password file <none>
I20260812 11:15:21.556679 4651 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:21.556797 4651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:21.557051 4651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:21.559926 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "fcae460f067e4ecd92cff48e2bd0770d"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-7z32"
server_key: "90105387d4192137a9b4d58ea25f1198"
server_key_iv: "40f0088f654c297c284a100083b55449"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.560668 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "fcae460f067e4ecd92cff48e2bd0770d"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-7z32"
server_key: "90105387d4192137a9b4d58ea25f1198"
server_key_iv: "40f0088f654c297c284a100083b55449"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.566228 4651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:21.570201 4672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.572248 4651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:21.572485 4651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "fcae460f067e4ecd92cff48e2bd0770d"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-7z32"
server_key: "90105387d4192137a9b4d58ea25f1198"
server_key_iv: "40f0088f654c297c284a100083b55449"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.572644 4651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:21.597733 4651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:21.602275 4651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:21.602578 4651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:21.603399 4651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:21.604561 4651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:21.604648 4651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.604746 4651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:21.604797 4651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.617761 4651 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38559
I20260812 11:15:21.617844 4850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38559 every 8 connection(s)
I20260812 11:15:21.619028 4651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:21.628583 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 4651
I20260812 11:15:21.628738 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:21.629093 28084 external_mini_cluster.cc:1428] Setting key ba3a79adfe330b1d839effa488753bb2
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:21.639403 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.621948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:59827 (local address 127.27.109.62:44685)
0812 11:15:21.622194 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.622201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.623909 (+ 1708us) server_negotiation.cc:407] Connection header received
0812 11:15:21.625915 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.625919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.625997 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.626121 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.628134 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.628989 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.631266 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.631575 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.631683 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.635299 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.635304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.635306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:21.635335 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.636893 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.637520 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.637529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.637530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.637611 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.638027 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.638063 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.638064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.638287 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.638404 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.639083 (+ 679us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.639229 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260812 11:15:21.640434 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.622659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44685 (local address 127.27.109.3:59827)
0812 11:15:21.623642 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.623728 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.625542 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.626401 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.626412 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.627122 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:21.627868 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.627886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.629200 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.629206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.631106 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.631122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.631796 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.631801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.632066 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.633066 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:21.633092 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:21.635050 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:21.637069 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:21.637075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:21.637093 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:21.637391 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:21.637713 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:21.637716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:21.637718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:21.637883 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:21.638404 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:21.638412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:21.638737 (+ 325us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.639024 (+ 287us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.639414 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":191,"threads_started":1}
I20260812 11:15:21.642395 4851 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44685
I20260812 11:15:21.642809 4851 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:21.643716 4851 heartbeater.cc:511] Master 127.27.109.62:44685 requested a full tablet report, sending...
I20260812 11:15:21.645349 4137 ts_manager.cc:194] Registered new tserver with Master: fcae460f067e4ecd92cff48e2bd0770d (127.27.109.3:38559)
I20260812 11:15:21.646175 4137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:59827
I20260812 11:15:21.658344 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/test-admin.keytab.
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.685559 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 11:15:21.689788 4866 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:21.687819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.27.109.1:36785)
0812 11:15:21.688281 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.688289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.688323 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:15:21.688409 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.688418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.688661 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.688836 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.689583 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60926: BlockingRecv error: recv got EOF from 127.0.0.1:60926 (error 108)
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":139,"threads_started":1}
W20260812 11:15:21.690541 28084 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.692230 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.693571 28084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:15:21.706698 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.694971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.27.109.1:36785)
0812 11:15:21.695156 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.695162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.695302 (+ 140us) server_negotiation.cc:407] Connection header received
0812 11:15:21.695565 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.695569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.695633 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.695731 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.696402 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.697282 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.698281 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.698737 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.698906 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.702093 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.702105 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.702122 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:21.702149 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.704480 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.705064 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.705071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.705072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.705169 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.705501 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.705505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.705507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.705837 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.706103 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.706354 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.706538 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 11:15:21.712196 4866 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:21.710715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.27.109.1:36785)
0812 11:15:21.710924 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.710930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.711274 (+ 344us) server_negotiation.cc:407] Connection header received
0812 11:15:21.711347 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.711350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.711410 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.711516 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.712039 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60942: BlockingRecv error: recv got EOF from 127.0.0.1:60942 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 11:15:21.712352 28084 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.713680 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.714795 28084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:15:21.726157 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.716005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.27.109.1:36785)
0812 11:15:21.716185 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.716190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.716206 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:15:21.716357 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.716359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.716406 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.716485 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.717164 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.717865 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.718951 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.719230 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.719355 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.722351 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.722356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.722358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:21.722426 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.724189 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.724730 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.724733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.724735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.724822 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.725113 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.725116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.725118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.725358 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.725615 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.725765 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.725943 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260812 11:15:21.730248 28084 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533282802791-28084-0/krb5kdc/krb5cc' of type FILE
W20260812 11:15:21.730262 4866 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:21.728693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.27.109.1:36785)
0812 11:15:21.728927 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.728932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.728953 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:15:21.729150 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.729154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.729208 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.729354 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.730100 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60958: BlockingRecv error: recv got EOF from 127.0.0.1:60958 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.731674 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:21.732651 28084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:15:21.744588 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.733745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.27.109.1:36785)
0812 11:15:21.733957 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.733962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.734079 (+ 117us) server_negotiation.cc:407] Connection header received
0812 11:15:21.734201 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.734204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.734263 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.734354 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.735238 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.735867 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.736997 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.737224 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.737369 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.740618 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.740622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.740626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:21.740663 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.742336 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.742877 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.742881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.742885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.742999 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:21.743363 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:21.743366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:21.743368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:21.743592 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.743757 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.744129 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.744387 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 11:15:21.746166 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 4240
I20260812 11:15:21.775394 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 4445
I20260812 11:15:21.790215 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 4651
I20260812 11:15:21.804121 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 4085
2026-08-12T11:15:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3216 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4886](info): set up 2 sockets
Aug 12 11:15:21 dist-test-slave-7z32 krb5kdc[4886](info): commencing operation
krb5kdc: starting...
W20260812 11:15:23.217561 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.371s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:23 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
I20260812 11:15:23.387925 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34707
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:23.543390 4903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:23.543848 4903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:23.543898 4903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:23.548533 4903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:23.548645 4903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:23.548666 4903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:23.548688 4903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:23.548705 4903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:23.555792 4903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:23.557497 4903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:23.558990 4903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:23.571581 4913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:23.571504 4914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:23.571504 4916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:23.572005 4903 server_base.cc:1034] running on GCE node
I20260812 11:15:23.572857 4903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:23.574522 4903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:23.575803 4903 hybrid_clock.cc:648] HybridClock initialized: now 1786533323575775 us; error 55 us; skew 500 ppm
Aug 12 11:15:23 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:23.580804 4903 init.cc:377] Logged in from keytab as oryx/127.27.109.62@KRBTEST.COM (short username oryx)
I20260812 11:15:23.583115 4903 webserver.cc:460] Webserver started at http://127.27.109.62:36369/ using document root <none> and password file <none>
I20260812 11:15:23.584028 4903 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:23.584125 4903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:23.584455 4903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:23.587616 4903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "ac156fbd8c8f48938db03a94ffeea18e"
format_stamp: "Formatted at 2026-08-12 11:15:23 on dist-test-slave-7z32"
server_key: "7db4db11ac26442d58f70ae5d0f0347e"
server_key_iv: "924c475f0ef511f2c832547130445512"
server_key_version: "encryptionkey@0"
I20260812 11:15:23.588476 4903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "ac156fbd8c8f48938db03a94ffeea18e"
format_stamp: "Formatted at 2026-08-12 11:15:23 on dist-test-slave-7z32"
server_key: "7db4db11ac26442d58f70ae5d0f0347e"
server_key_iv: "924c475f0ef511f2c832547130445512"
server_key_version: "encryptionkey@0"
I20260812 11:15:23.594134 4903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:15:23.598194 4928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:23.600376 4903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 11:15:23.600652 4903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "ac156fbd8c8f48938db03a94ffeea18e"
format_stamp: "Formatted at 2026-08-12 11:15:23 on dist-test-slave-7z32"
server_key: "7db4db11ac26442d58f70ae5d0f0347e"
server_key_iv: "924c475f0ef511f2c832547130445512"
server_key_version: "encryptionkey@0"
I20260812 11:15:23.600873 4903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:23.614985 4903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:23.620189 4903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:23.620481 4903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:23.631245 4903 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:34707
I20260812 11:15:23.631657 5013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:34707 every 8 connection(s)
I20260812 11:15:23.632673 4903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:23.636260 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:23.639958 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 4903
I20260812 11:15:23.640125 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:23.640502 28084 external_mini_cluster.cc:1428] Setting key 579ef13b860c6e0772dd20cffada1e54
I20260812 11:15:23.643782 5015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: Bootstrap starting.
I20260812 11:15:23.646970 5015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:23.647939 5015 log.cc:824] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:23.651216 5015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: No bootstrap required, opened a new log
Aug 12 11:15:23 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:23.656561 5015 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } }
I20260812 11:15:23.656920 5015 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:23.656991 5015 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac156fbd8c8f48938db03a94ffeea18e, State: Initialized, Role: FOLLOWER
I20260812 11:15:23.657593 5015 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } }
I20260812 11:15:23.657779 5015 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:23.657853 5015 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:23.657982 5015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:23.659410 5015 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } }
I20260812 11:15:23.659852 5015 leader_election.cc:302] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac156fbd8c8f48938db03a94ffeea18e; no voters:
I20260812 11:15:23.660279 5015 leader_election.cc:288] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:23.660862 5025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:23.661273 5025 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 1 LEADER]: Becoming Leader. State: Replica: ac156fbd8c8f48938db03a94ffeea18e, State: Running, Role: LEADER
I20260812 11:15:23.661779 5015 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:23.661741 5025 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } }
I20260812 11:15:23.661753 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:23.642675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.27.109.62:34707)
0812 11:15:23.643421 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:23.643444 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:23.643497 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:15:23.644537 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:23.644565 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:23.645069 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:23.645517 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:23.646727 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:23.648612 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:23.650322 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:23.651156 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:23.651415 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:23.655005 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:23.655014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:23.655039 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:23.655090 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:23.657410 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:23.658314 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:23.658322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:23.658329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:23.658492 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:23.658934 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:23.658937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:23.658939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:23.659597 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:23.659802 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:23.660238 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 11:15:23.660618 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":207,"threads_started":1}
I20260812 11:15:23.664206 5029 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac156fbd8c8f48938db03a94ffeea18e. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } }
I20260812 11:15:23.664394 5029 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:23.664743 5028 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } }
I20260812 11:15:23.665894 5028 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:23.669821 5048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:23.669960 5048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:23.670042 5049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:23.671000 5049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:23.678611 5049 catalog_manager.cc:1347] Generated new cluster ID: 5561c6c2e0b14a8ca990e91eb3464e52
I20260812 11:15:23.678766 5049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:23.693850 5049 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:23.695333 5049 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:23.707053 5049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: Generated new TSK 0
I20260812 11:15:23.708616 5049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
I20260812 11:15:23.760494 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34707
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:23.914279 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:23.914795 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:23.914943 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:23.920411 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:23.920627 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:23.920764 5066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:23.927721 5066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:23.929401 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:23.930948 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:23.945124 5077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:23.945365 5066 server_base.cc:1034] running on GCE node
W20260812 11:15:23.945192 5078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:23.945464 5080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:23.946190 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:23.947273 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:23.948535 5066 hybrid_clock.cc:648] HybridClock initialized: now 1786533323948504 us; error 50 us; skew 500 ppm
W20260812 11:15:24.750371 5066 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-7z32': real 0.801s user 0.001s sys 0.000s
Aug 12 11:15:24 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533324, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:24.753707 5066 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260812 11:15:24.755311 5066 webserver.cc:460] Webserver started at http://127.27.109.1:33663/ using document root <none> and password file <none>
I20260812 11:15:24.756132 5066 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:24.756250 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:24.756559 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:24.759559 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "18168ee7cdda412a90889a1be0714fc2"
format_stamp: "Formatted at 2026-08-12 11:15:24 on dist-test-slave-7z32"
server_key: "49c975fac6f234a7cd543ce8936de209"
server_key_iv: "0857b4260537801c499fd56d0e38f757"
server_key_version: "encryptionkey@0"
I20260812 11:15:24.760394 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "18168ee7cdda412a90889a1be0714fc2"
format_stamp: "Formatted at 2026-08-12 11:15:24 on dist-test-slave-7z32"
server_key: "49c975fac6f234a7cd543ce8936de209"
server_key_iv: "0857b4260537801c499fd56d0e38f757"
server_key_version: "encryptionkey@0"
I20260812 11:15:24.766518 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:15:24.770742 5099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:24.772724 5066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:24.772943 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "18168ee7cdda412a90889a1be0714fc2"
format_stamp: "Formatted at 2026-08-12 11:15:24 on dist-test-slave-7z32"
server_key: "49c975fac6f234a7cd543ce8936de209"
server_key_iv: "0857b4260537801c499fd56d0e38f757"
server_key_version: "encryptionkey@0"
I20260812 11:15:24.773149 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:24.793056 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:24.798686 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:24.799077 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:24.800087 5066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:24.801578 5066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:24.801671 5066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:24.801774 5066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:24.801822 5066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:24.816702 5066 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37737
I20260812 11:15:24.816783 5280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37737 every 8 connection(s)
I20260812 11:15:24.818109 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:24.822947 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 5066
I20260812 11:15:24.823138 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:24.823532 28084 external_mini_cluster.cc:1428] Setting key 63e35fd0ecd81e8de77e16c2b947c823
Aug 12 11:15:24 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533324, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:24.837015 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:24.821065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33285 (local address 127.27.109.62:34707)
0812 11:15:24.821521 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:24.821526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:24.822670 (+ 1144us) server_negotiation.cc:407] Connection header received
0812 11:15:24.824761 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:24.824765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:24.824842 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:24.824950 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:24.826565 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:24.827248 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:24.828539 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:24.828781 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:24.828901 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:24.831828 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:24.831832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:24.831834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:24.831861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:24.834142 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:24.834915 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:24.834920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:24.834924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:24.835048 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:24.835521 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:24.835524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:24.835526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:24.835763 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:24.835906 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:24.836583 (+ 677us) server_negotiation.cc:299] Negotiation successful
0812 11:15:24.836772 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":336,"thread_start_us":166,"threads_started":1}
I20260812 11:15:24.838130 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:24.821704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:33285)
0812 11:15:24.822415 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:24.822492 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:24.824471 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:24.825190 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:24.825198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:24.825777 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:24.826388 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:24.826404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:24.827406 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:24.827410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:24.828386 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:24.828397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:24.828963 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:24.828967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:24.829154 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:24.829862 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:24.829881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:24.831612 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:24.834343 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:24.834352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:24.834381 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:24.834751 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:24.835176 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:24.835180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:24.835182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:24.835379 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:24.835906 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:24.835917 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:24.836258 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 11:15:24.836513 (+ 255us) client_negotiation.cc:240] Negotiation successful
0812 11:15:24.836983 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for oryx/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:15:24.840965 5281 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34707
I20260812 11:15:24.841394 5281 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:24.842142 5281 heartbeater.cc:511] Master 127.27.109.62:34707 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:24.845966 4954 ts_manager.cc:194] Registered new tserver with Master: 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737)
I20260812 11:15:24.847805 4954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:33285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
I20260812 11:15:24.868227 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34707
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:25.062369 5291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:25.063063 5291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:25.063144 5291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:25.068562 5291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:25.068689 5291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:25.068765 5291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:25.075986 5291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:25.077679 5291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:25.079263 5291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:25.092832 5300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:25.092855 5299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:25.092844 5302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:25.093494 5291 server_base.cc:1034] running on GCE node
I20260812 11:15:25.094087 5291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:25.095048 5291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:25.096464 5291 hybrid_clock.cc:648] HybridClock initialized: now 1786533325096281 us; error 58 us; skew 500 ppm
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:25.100359 5291 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260812 11:15:25.101527 5291 webserver.cc:460] Webserver started at http://127.27.109.2:41491/ using document root <none> and password file <none>
I20260812 11:15:25.102246 5291 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:25.102329 5291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:25.102593 5291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:25.105120 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "0812419ccd80493bb22afa94d94683f6"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "ff284e240f432fbca2c517dc91b26b8f"
server_key_iv: "cf5e0ac4b7056f8723eb8a7bf026775c"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.105881 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "0812419ccd80493bb22afa94d94683f6"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "ff284e240f432fbca2c517dc91b26b8f"
server_key_iv: "cf5e0ac4b7056f8723eb8a7bf026775c"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.110952 5291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 11:15:25.114463 5315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.116895 5291 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:25.117103 5291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "0812419ccd80493bb22afa94d94683f6"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "ff284e240f432fbca2c517dc91b26b8f"
server_key_iv: "cf5e0ac4b7056f8723eb8a7bf026775c"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.117317 5291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:25.152972 5291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:25.157186 5291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:25.157474 5291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:25.158298 5291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:25.159390 5291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:25.159474 5291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.159557 5291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:25.159603 5291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.172456 5291 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:39669
I20260812 11:15:25.172518 5508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:39669 every 8 connection(s)
I20260812 11:15:25.173643 5291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:25.182461 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 5291
I20260812 11:15:25.182662 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:25.183086 28084 external_mini_cluster.cc:1428] Setting key d502640e2569059688ef3df6bb9841a5
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:25.195662 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.176766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:45587 (local address 127.27.109.62:34707)
0812 11:15:25.176998 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.177003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.178520 (+ 1517us) server_negotiation.cc:407] Connection header received
0812 11:15:25.180487 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.180491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.180581 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.180711 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.182812 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.183790 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.186003 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.186431 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.186587 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.190014 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.190019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.190021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.190059 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.191995 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.192974 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.192977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.192978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.193082 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.193627 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.193630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.193632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.193871 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.193967 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.195263 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.195498 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 11:15:25.197450 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.177308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:45587)
0812 11:15:25.178103 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.178177 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.180139 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.181000 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.181011 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.181570 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.182451 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.182480 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.184061 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.184065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.185707 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.185731 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.186648 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.186651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.186968 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.187820 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.187842 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.189692 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.192179 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.192192 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.192224 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.192819 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.193207 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.193212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.193214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.193483 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.194016 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.194030 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.194609 (+ 579us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.195198 (+ 589us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.195833 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":165,"threads_started":1}
WARNING: no policy specified for oryx/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:15:25.199328 5509 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34707
I20260812 11:15:25.199863 5509 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:25.200649 5509 heartbeater.cc:511] Master 127.27.109.62:34707 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.3@KRBTEST.COM" created.
I20260812 11:15:25.202441 4954 ts_manager.cc:194] Registered new tserver with Master: 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:25.203241 4954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:45587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
I20260812 11:15:25.224082 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34707
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:25.381332 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:25.381800 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:25.381865 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:25.387887 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:25.388084 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:25.388176 5518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:25.396250 5518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:25.397931 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:25.399405 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:25.412833 5530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:25.412806 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:25.413058 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:25.413460 5518 server_base.cc:1034] running on GCE node
I20260812 11:15:25.414124 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:25.415262 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:25.416481 5518 hybrid_clock.cc:648] HybridClock initialized: now 1786533325416437 us; error 55 us; skew 500 ppm
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:25.420462 5518 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260812 11:15:25.421833 5518 webserver.cc:460] Webserver started at http://127.27.109.3:39673/ using document root <none> and password file <none>
I20260812 11:15:25.422572 5518 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:25.422646 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:25.422952 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:25.425812 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "81e79624a9d040aa88f9ae538cc70621"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "6a91e227da3468d946995b1e4bc4ef1d"
server_key_iv: "b9ed85e7b1013a06b8bcb077df796b8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.426506 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "81e79624a9d040aa88f9ae538cc70621"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "6a91e227da3468d946995b1e4bc4ef1d"
server_key_iv: "b9ed85e7b1013a06b8bcb077df796b8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.432004 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:15:25.435966 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.437985 5518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:25.438226 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "81e79624a9d040aa88f9ae538cc70621"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "6a91e227da3468d946995b1e4bc4ef1d"
server_key_iv: "b9ed85e7b1013a06b8bcb077df796b8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.438432 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:25.469344 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:25.474462 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:25.474849 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:25.475730 5518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:25.476881 5518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:25.476964 5518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.477056 5518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:25.477104 5518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.490975 5518 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:46381
I20260812 11:15:25.491115 5710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:46381 every 8 connection(s)
I20260812 11:15:25.492449 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:25.497061 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 5518
I20260812 11:15:25.497238 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:25.497598 28084 external_mini_cluster.cc:1428] Setting key 40bbc80df01e42f36cb3713461eec537
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:25.512188 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.495320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:33623 (local address 127.27.109.62:34707)
0812 11:15:25.495488 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.495493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.496734 (+ 1241us) server_negotiation.cc:407] Connection header received
0812 11:15:25.498811 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.498816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.498926 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.499026 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.500946 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.501601 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.503063 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.503385 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.503529 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.507172 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.507177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.507179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.507235 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.508917 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.509703 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.509706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.509708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.509795 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.510323 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.510326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.510328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.510567 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.510666 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.511754 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.511949 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:15:25.513357 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.495825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:33623)
0812 11:15:25.496501 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.496565 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.498475 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.499342 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.499353 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.499968 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.500744 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.500765 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.501777 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.501780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.502777 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.502787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.503640 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.503644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.503825 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.504656 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.504676 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.506913 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.509126 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.509134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.509154 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.509570 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.509932 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.509935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.509937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.510155 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.510740 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.510748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.511301 (+ 553us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.511597 (+ 296us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.512130 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":168,"threads_started":1}
I20260812 11:15:25.514930 5711 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34707
I20260812 11:15:25.515318 5711 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:25.516055 5711 heartbeater.cc:511] Master 127.27.109.62:34707 requested a full tablet report, sending...
I20260812 11:15:25.517374 4954 ts_manager.cc:194] Registered new tserver with Master: 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381)
I20260812 11:15:25.518023 4954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:33623
I20260812 11:15:25.525905 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.61.keytab.
Entry for principal oryx/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.61.keytab.
I20260812 11:15:25.563441 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:39379
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.61.keytab
--principal=oryx/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34707,127.27.109.61:39379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:25.704013 5723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:25.704411 5723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:25.704460 5723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:25.709296 5723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:25.709396 5723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:25.709417 5723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:25.709436 5723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:25.709453 5723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:25.715778 5723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34707,127.27.109.61:39379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.61.keytab
--principal=oryx/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
W20260812 11:15:25.716079 5723 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.109.62:34707,127.27.109.61:39379'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 11:15:25.717133 5723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:25.718279 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:25.730228 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:25.730355 5723 server_base.cc:1034] running on GCE node
W20260812 11:15:25.730198 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:25.730201 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:25.731441 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:25.732790 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:25.734030 5723 hybrid_clock.cc:648] HybridClock initialized: now 1786533325733991 us; error 73 us; skew 500 ppm
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:25.737561 5723 init.cc:377] Logged in from keytab as oryx/127.27.109.61@KRBTEST.COM (short username oryx)
I20260812 11:15:25.738976 5723 webserver.cc:460] Webserver started at http://127.27.109.61:41703/ using document root <none> and password file <none>
I20260812 11:15:25.739709 5723 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:25.739838 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:25.740159 5723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:25.742641 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/instance:
uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "1af3b5499e18d82b9188ce4cb0146bf3"
server_key_iv: "000d394764e8e18d80a558f9ec522a01"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.743444 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal/instance:
uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "1af3b5499e18d82b9188ce4cb0146bf3"
server_key_iv: "000d394764e8e18d80a558f9ec522a01"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.748592 5723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260812 11:15:25.752403 5744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.754311 5723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 11:15:25.754551 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "1af3b5499e18d82b9188ce4cb0146bf3"
server_key_iv: "000d394764e8e18d80a558f9ec522a01"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.754755 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:25.784132 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:25.788134 5723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:25.788508 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:25.797469 5723 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:39379
I20260812 11:15:25.798018 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:39379 every 8 connection(s)
I20260812 11:15:25.798755 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
I20260812 11:15:25.803206 5821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:25.806248 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 5723
I20260812 11:15:25.810094 5821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:25.825985 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.813304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.27.109.62:34707)
0812 11:15:25.813469 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.813472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.813991 (+ 519us) server_negotiation.cc:407] Connection header received
0812 11:15:25.814225 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.814228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.814280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.814375 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.815998 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.816686 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.817880 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.818139 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.818282 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.821715 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.821719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.821721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.821758 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.823522 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.824229 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.824233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.824234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.824325 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.824738 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.824741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.824742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.825022 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.825112 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.825607 (+ 495us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.825787 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:15:25.827515 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.813200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:34424)
0812 11:15:25.813822 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.813879 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.814082 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.815001 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.815009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.815655 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.815834 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.815842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.816938 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.816943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.817703 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.817714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.818328 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.818333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.818448 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.819580 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.819604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.821516 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.823656 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.823665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.823681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.824099 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.824456 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.824462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.824464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.824600 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.825153 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.825161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.825319 (+ 158us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.825544 (+ 225us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.826073 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":277,"threads_started":1}
I20260812 11:15:25.827515 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.808798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.27.109.61:39379)
0812 11:15:25.809386 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.809418 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.809466 (+ 48us) server_negotiation.cc:407] Connection header received
0812 11:15:25.810316 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.810336 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.810746 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.811226 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.812490 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.814245 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.815560 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.816137 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.816465 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.820409 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.820418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.820440 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.820487 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.823397 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.824164 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.824174 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.824178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.824333 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.824665 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.824671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.824674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.825304 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.825589 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.825831 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.826378 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":341,"server-negotiator.queue_time_us":402,"thread_start_us":203,"threads_started":1}
I20260812 11:15:25.829555 5821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260812 11:15:25.840659 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.830353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.27.109.61:39379)
0812 11:15:25.830509 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.830514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.830534 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:15:25.830732 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.830735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.830789 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.830951 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.831459 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.832859 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.833796 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.834090 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.834215 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.836913 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.836917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.836919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.836949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.838682 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.839237 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.839242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.839244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.839328 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.839628 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.839638 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.839640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.839863 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.839976 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.840256 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.840432 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":63}
I20260812 11:15:25.840659 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.830207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.0.0.1:47842)
0812 11:15:25.830468 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.830483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.830597 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.831000 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.831003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.831189 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.831307 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.831313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.833013 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.833016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.833640 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.833649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.834232 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.834234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.834387 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.835069 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.835083 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.836720 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.838852 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.838858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.838861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.839085 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.839426 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.839431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.839433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.839521 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.839961 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.839964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.840059 (+ 95us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.840183 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.840376 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":44}
I20260812 11:15:25.845223 5821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Bootstrap starting.
I20260812 11:15:25.848069 5821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:25.848894 5821 log.cc:824] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:25.852111 5821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: No bootstrap required, opened a new log
I20260812 11:15:25.852075 5281 heartbeater.cc:503] Master 127.27.109.62:34707 was elected leader, sending a full tablet report...
I20260812 11:15:25.856272 5821 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } }
I20260812 11:15:25.856546 5821 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:25.856585 5821 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5eb9c6a7f4644129c9c0524b6bb5d3a, State: Initialized, Role: FOLLOWER
I20260812 11:15:25.857029 5821 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } }
I20260812 11:15:25.857717 5830 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } } }
I20260812 11:15:25.857956 5830 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:25.858619 5821 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: configured and running, proceeding with master startup.
W20260812 11:15:25.865859 5850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:25.866021 5850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:25.867970 5723 master_runner.cc:428] Detected that this master e5eb9c6a7f4644129c9c0524b6bb5d3a is joining an existing cluster
I20260812 11:15:25.868039 5723 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.109.61:39379
I20260812 11:15:25.881091 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.871029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.27.109.61:39379)
0812 11:15:25.871169 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.871175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.871943 (+ 768us) server_negotiation.cc:407] Connection header received
0812 11:15:25.872151 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.872157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.872207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.872307 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.872985 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.873784 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.875078 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.875329 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.875462 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.876645 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.876649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.876653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.876675 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.879012 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.879494 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.879500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.879502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.879591 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.879899 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.879902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.879904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.880187 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.880295 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.880635 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.880878 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":55}
I20260812 11:15:25.881609 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.871083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.27.109.62:34707)
0812 11:15:25.871265 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.871270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.871571 (+ 301us) server_negotiation.cc:407] Connection header received
0812 11:15:25.871729 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.871733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.871793 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.871898 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.872711 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.873289 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.874312 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.874544 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.874662 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.876713 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.876718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.876721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.876772 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.879376 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.880043 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.880047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.880049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.880171 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.880558 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.880564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.880566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.880839 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.880948 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.881308 (+ 360us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.881454 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:15:25.881091 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.871371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.0.0.1:47850)
0812 11:15:25.871838 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.871855 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.872023 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.872371 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.872374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.872679 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.872847 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.872855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.873928 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.873931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.874930 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.874942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.875463 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.875467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.875566 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.876037 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.876056 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.876489 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.879143 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.879146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.879149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.879384 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.879692 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.879695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.879697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.879800 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.880329 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.880332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.880441 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.880570 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.880786 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"spinlock_wait_cycles":1792,"thread_start_us":129,"threads_started":1}
I20260812 11:15:25.882309 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.870897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:34440)
0812 11:15:25.871445 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.871460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.871618 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.872071 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.872077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.872428 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.872555 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.872562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.873516 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.873521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.874169 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.874177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.874685 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.874689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.874789 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.875789 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.875801 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.876489 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.879577 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.879582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.879585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.879900 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.880300 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.880303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.880305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.880426 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.881000 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.881003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.881097 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.881908 (+ 811us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.882092 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":150,"threads_started":1}
WARNING: no policy specified for oryx/127.27.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.60@KRBTEST.COM" created.
Aug 12 11:15:25 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260812 11:15:25.912014 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.890246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.0.0.1:47856)
0812 11:15:25.894030 (+ 3784us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.894136 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.894601 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.895164 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.895174 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.895805 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.895978 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.895986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.896857 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.896861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.897636 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.897647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.898270 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.898275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.898420 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.899267 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.899281 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.905912 (+ 6631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.908763 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.908784 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.908795 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.909261 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.909625 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.909628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.909630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.909727 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.910540 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.910551 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.910792 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.911393 (+ 601us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.911765 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3281,"thread_start_us":153,"threads_started":1}
I20260812 11:15:25.912046 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.890274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.27.109.61:39379)
0812 11:15:25.890458 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.890465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.894417 (+ 3952us) server_negotiation.cc:407] Connection header received
0812 11:15:25.894885 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.894889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.894962 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.895080 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.896149 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.896717 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.897817 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.898083 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.898253 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.906272 (+ 8019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.906278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.906282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.906335 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.908526 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.909403 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.909408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.909410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.909505 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.909906 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.909915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.909920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.910337 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.910504 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.911439 (+ 935us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.911740 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:15:25.915313 4954 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.109.61:39379: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
member_type: NON_VOTER
last_known_addr {
host: "127.27.109.61"
port: 39379
}
attrs {
promote: true
}
}
dest_uuid: "ac156fbd8c8f48938db03a94ffeea18e"
cas_config_opid_index: -1
I20260812 11:15:25.916353 4954 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } }
I20260812 11:15:25.919335 5872 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } }
I20260812 11:15:25.920859 5781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:25.922084 5872 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.109.61:39379
I20260812 11:15:25.922956 5723 master.cc:537] Master@127.27.109.61:39379 shutting down...
I20260812 11:15:25.924369 5781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Refusing update from remote peer ac156fbd8c8f48938db03a94ffeea18e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 11:15:25.924428 5869 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } }
I20260812 11:15:25.924578 5869 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:25.926702 5869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:15:25.930035 4930 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 11:15:25.931449 5723 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 11:15:25.931478 4930 consensus_peers.cc:487] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e -> Peer e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61:39379): Couldn't send request to peer e5eb9c6a7f4644129c9c0524b6bb5d3a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 11:15:25.931903 5723 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:25.932053 5723 tablet_replica.cc:330] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.60.keytab.
Entry for principal oryx/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.60.keytab.
I20260812 11:15:25.934680 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:45539
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.60.keytab
--principal=oryx/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:25.940402 5723 master.cc:552] Master@127.27.109.61:39379 shutdown complete.
I20260812 11:15:25.940606 5723 master_runner.cc:305] Clearing existing system tablet
I20260812 11:15:25.940992 5723 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:25.941133 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:25.943385 5723 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:25.944896 5878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.945327 5723 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:25.945746 5723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:25.945919 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "1af3b5499e18d82b9188ce4cb0146bf3"
server_key_iv: "000d394764e8e18d80a558f9ec522a01"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.949777 5723 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:25.954757 5723 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 11:15:25.955143 5723 log.cc:1196] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 11:15:25.955605 5723 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Deleting consensus metadata
I20260812 11:15:25.955874 5723 master_runner.cc:315] Copying system tablet from 127.27.109.62:34707
I20260812 11:15:25.957804 5723 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.109.62:34707
I20260812 11:15:25.970597 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.958803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:34456)
0812 11:15:25.959385 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:25.959405 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:25.959560 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:25.959987 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:25.959991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:25.960301 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:25.960476 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.960487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.961760 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.961764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.962985 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:25.963000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.963877 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.963884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:25.964009 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.964831 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:25.964857 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:25.965387 (+ 530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:25.968187 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.968191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.968194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.968556 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.969005 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:25.969009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:25.969011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:25.969153 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:25.969858 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:25.969862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:25.970002 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 11:15:25.970145 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 11:15:25.970338 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":136,"threads_started":1}
I20260812 11:15:25.970659 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.959102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.27.109.62:34707)
0812 11:15:25.959304 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.959310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.959522 (+ 212us) server_negotiation.cc:407] Connection header received
0812 11:15:25.959698 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.959702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.959776 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.959941 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.960669 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.961610 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.963166 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.963576 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.963697 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.965623 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.965629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.965632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:25.965673 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.968025 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.968734 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.968740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.968743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.968875 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:25.969383 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:25.969387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:25.969389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:25.969709 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.969848 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.970241 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.970420 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 11:15:25.972551 4985 tablet_copy_service.cc:140] P ac156fbd8c8f48938db03a94ffeea18e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e5eb9c6a7f4644129c9c0524b6bb5d3a ({username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:34456)
I20260812 11:15:25.972817 4985 tablet_copy_service.cc:161] P ac156fbd8c8f48938db03a94ffeea18e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e5eb9c6a7f4644129c9c0524b6bb5d3a at {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:34456: session id = e5eb9c6a7f4644129c9c0524b6bb5d3a-00000000000000000000000000000000
I20260812 11:15:25.975416 4985 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 11:15:25.978603 5723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 11:15:25.982026 5723 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: tablet copy: Starting download of 0 data blocks...
I20260812 11:15:25.982331 5723 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: tablet copy: Starting download of 1 WAL segments...
I20260812 11:15:25.985872 5723 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 11:15:25.989308 4985 tablet_copy_service.cc:342] P ac156fbd8c8f48938db03a94ffeea18e: Request end of tablet copy session e5eb9c6a7f4644129c9c0524b6bb5d3a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:34456
I20260812 11:15:25.989563 4985 tablet_copy_service.cc:433] P ac156fbd8c8f48938db03a94ffeea18e: ending tablet copy session e5eb9c6a7f4644129c9c0524b6bb5d3a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e5eb9c6a7f4644129c9c0524b6bb5d3a
I20260812 11:15:26.006501 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:26.009908 5893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.010247 5887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.009735 5889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:26.012240 5723 server_base.cc:1034] running on GCE node
I20260812 11:15:26.012615 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:26.014364 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:26.015656 5723 hybrid_clock.cc:648] HybridClock initialized: now 1786533326015623 us; error 60 us; skew 500 ppm
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:26.018771 5723 init.cc:377] Logged in from keytab as oryx/127.27.109.61@KRBTEST.COM (short username oryx)
I20260812 11:15:26.019595 5723 webserver.cc:460] Webserver started at http://127.27.109.61:34829/ using document root <none> and password file <none>
I20260812 11:15:26.019855 5723 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:26.019918 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:26.022089 5723 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:26.025256 5906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:26.026432 5723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:15:26.027143 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "1af3b5499e18d82b9188ce4cb0146bf3"
server_key_iv: "000d394764e8e18d80a558f9ec522a01"
server_key_version: "encryptionkey@0"
I20260812 11:15:26.027516 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:26.046743 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:26.049853 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:26.057333 5723 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:39379
I20260812 11:15:26.058305 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
I20260812 11:15:26.062521 5987 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:15:26.064981 5987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Bootstrap starting.
I20260812 11:15:26.068861 5986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:39379 every 8 connection(s)
I20260812 11:15:26.086282 5987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:15:26.087240 5987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: Bootstrap complete.
I20260812 11:15:26.088747 5987 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } }
I20260812 11:15:26.089184 5987 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5eb9c6a7f4644129c9c0524b6bb5d3a, State: Initialized, Role: LEARNER
I20260812 11:15:26.089423 5987 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } }
I20260812 11:15:26.090567 5987 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:26.095604 5991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } }
I20260812 11:15:26.095883 5991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: LEARNER
I20260812 11:15:26.097628 6003 catalog_manager.cc:1259] Loaded cluster ID: 5561c6c2e0b14a8ca990e91eb3464e52
I20260812 11:15:26.097710 6003 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: loading cluster ID for follower catalog manager: success
I20260812 11:15:26.101166 6003 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: acquiring CA information for follower catalog manager: success
I20260812 11:15:26.104270 6003 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 11:15:26.160204 5875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:26.160575 5875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:26.160626 5875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:26.165710 5875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:26.165813 5875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:26.165846 5875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:26.165866 5875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:26.165884 5875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:26.173588 5875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.60.keytab
--principal=oryx/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:45539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:26.175094 5875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:26.176592 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:26.192894 6013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.192823 6015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.192823 6012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:26.194015 5875 server_base.cc:1034] running on GCE node
I20260812 11:15:26.195142 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:26.196633 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:26.200184 5875 hybrid_clock.cc:648] HybridClock initialized: now 1786533326199917 us; error 131 us; skew 500 ppm
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:26.205372 5875 init.cc:377] Logged in from keytab as oryx/127.27.109.60@KRBTEST.COM (short username oryx)
I20260812 11:15:26.206014 5509 heartbeater.cc:503] Master 127.27.109.62:34707 was elected leader, sending a full tablet report...
I20260812 11:15:26.207434 5875 webserver.cc:460] Webserver started at http://127.27.109.60:35183/ using document root <none> and password file <none>
I20260812 11:15:26.208400 5875 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:26.208577 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:26.209103 5875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:26.213238 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/instance:
uuid: "b65a1cd651f84a9d9e0004637f6e5e1b"
format_stamp: "Formatted at 2026-08-12 11:15:26 on dist-test-slave-7z32"
server_key: "a7229c70fd33d9c41066c1f562b3673b"
server_key_iv: "39261516b275cd0fad662d3f2c5d02ed"
server_key_version: "encryptionkey@0"
I20260812 11:15:26.214532 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal/instance:
uuid: "b65a1cd651f84a9d9e0004637f6e5e1b"
format_stamp: "Formatted at 2026-08-12 11:15:26 on dist-test-slave-7z32"
server_key: "a7229c70fd33d9c41066c1f562b3673b"
server_key_iv: "39261516b275cd0fad662d3f2c5d02ed"
server_key_version: "encryptionkey@0"
I20260812 11:15:26.223464 5875 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 11:15:26.227892 6024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:26.230302 5875 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:15:26.230568 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
uuid: "b65a1cd651f84a9d9e0004637f6e5e1b"
format_stamp: "Formatted at 2026-08-12 11:15:26 on dist-test-slave-7z32"
server_key: "a7229c70fd33d9c41066c1f562b3673b"
server_key_iv: "39261516b275cd0fad662d3f2c5d02ed"
server_key_version: "encryptionkey@0"
I20260812 11:15:26.230800 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:26.256793 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:26.261446 5875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:26.261751 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:26.272810 5875 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:45539
I20260812 11:15:26.273315 6101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:45539 every 8 connection(s)
I20260812 11:15:26.274506 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
I20260812 11:15:26.278553 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:26.282480 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 5875
I20260812 11:15:26.284365 6103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:26.305955 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.288050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.27.109.62:34707)
0812 11:15:26.288237 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.288242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.289512 (+ 1270us) server_negotiation.cc:407] Connection header received
0812 11:15:26.289741 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.289745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.289828 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.289929 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.291353 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.292612 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.295187 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.295555 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.295745 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.299520 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.299529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.299536 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.299598 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.301857 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.302964 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.302978 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.302981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.303171 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.303812 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.303819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.303824 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.304223 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.304376 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.305379 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.305724 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:15:26.308475 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.285077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.27.109.60:45539)
0812 11:15:26.286027 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.286064 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.286128 (+ 64us) server_negotiation.cc:407] Connection header received
0812 11:15:26.287308 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.287337 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.288099 (+ 762us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.288670 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.289954 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.291724 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.293295 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.294161 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.294409 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.299094 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.299106 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.299135 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.299192 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.301827 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.303177 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.303191 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.303196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.303416 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.303938 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.303944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.303946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.304768 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.305323 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.305705 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.306741 (+ 1036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":707,"spinlock_wait_cycles":6784,"thread_start_us":391,"threads_started":1}
I20260812 11:15:26.308475 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.287721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:34460)
0812 11:15:26.289260 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.289346 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.289590 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.290114 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.290124 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.290764 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.291086 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.291109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.293124 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.293131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.294933 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.294960 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.295849 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.295855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.295996 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.296671 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.296693 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.299214 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.302079 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.302103 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.302126 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.302636 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.303318 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.303323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.303325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.303610 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.304395 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.304405 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.304768 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.306389 (+ 1621us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.306806 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"mutex_wait_us":207,"spinlock_wait_cycles":20096,"thread_start_us":190,"threads_started":1}
I20260812 11:15:26.311954 6103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260812 11:15:26.325003 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.312926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.0.0.1:47862)
0812 11:15:26.313113 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.313141 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.313242 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.313829 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.313833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.314106 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.314243 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.314249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.315731 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.315737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.316384 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.316394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.316959 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.316962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.317100 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.317774 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.317787 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.319434 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.322216 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.322232 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.322234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.323107 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.323550 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.323553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.323555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.323654 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.324283 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.324286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.324414 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.324535 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.324784 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 11:15:26.325022 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.313107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.27.109.61:39379)
0812 11:15:26.313537 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.313542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.313556 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:15:26.313614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.313617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.313668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.313799 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.314388 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.315571 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.316539 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.316809 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.316937 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.319650 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.319654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.319657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.319691 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.321905 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.323299 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.323303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.323304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.323423 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.323790 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.323793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.323794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.324145 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.324297 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.324615 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.324786 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":184,"threads_started":1}
I20260812 11:15:26.326696 6103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
I20260812 11:15:26.340204 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.327580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.0.0.1:33274)
0812 11:15:26.327888 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.327909 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.328045 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.328587 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.328591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.328860 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.328988 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.328994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.330233 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.330237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.331418 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.331436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.332190 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.332193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.332282 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.333092 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.333107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.335100 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.338014 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.338019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.338021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.338380 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.338799 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.338801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.338803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.338999 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.339532 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.339535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.339724 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.339835 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.340013 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260812 11:15:26.340248 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.327942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.27.109.60:45539)
0812 11:15:26.328162 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.328167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.328187 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:15:26.328236 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.328239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.328291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.328482 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.329182 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.329990 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.331641 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.331987 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.332110 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.335404 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.335409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.335412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.335468 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.337801 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.338549 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.338553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.338555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.338670 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.339147 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.339151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.339152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.339391 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.339497 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.339941 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.340086 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 11:15:26.344628 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Bootstrap starting.
I20260812 11:15:26.347313 6103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:26.348140 6103 log.cc:824] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:26.350901 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: No bootstrap required, opened a new log
I20260812 11:15:26.354276 6103 raft_consensus.cc:348] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } }
I20260812 11:15:26.354590 6103 raft_consensus.cc:374] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:26.354655 6103 raft_consensus.cc:729] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65a1cd651f84a9d9e0004637f6e5e1b, State: Initialized, Role: FOLLOWER
I20260812 11:15:26.355269 6103 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } }
I20260812 11:15:26.355877 6112 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } } }
I20260812 11:15:26.356031 6112 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:26.356792 6103 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:26.364166 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 5066
W20260812 11:15:26.385432 6126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:26.385596 6126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:26.389766 5875 master_runner.cc:428] Detected that this master b65a1cd651f84a9d9e0004637f6e5e1b is joining an existing cluster
I20260812 11:15:26.389873 5875 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.109.60:45539
I20260812 11:15:26.399436 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:37737
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:34707
--webserver_port=33663
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:26.425698 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.393270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.27.109.62:34707)
0812 11:15:26.393533 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.393541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.395050 (+ 1509us) server_negotiation.cc:407] Connection header received
0812 11:15:26.395301 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.395305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.395367 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.395428 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.396290 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.397156 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.398622 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.399036 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.399115 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.400859 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.400864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.400867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.400893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.402670 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.403265 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.403269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.403271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.403364 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.423648 (+ 20284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.423654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.423656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.424011 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.424122 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.425308 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.425498 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 11:15:26.426313 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.393727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:34462)
0812 11:15:26.394933 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.394948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.395144 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.395602 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.395605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.395968 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.396121 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.396129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.397321 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.397324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.398468 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.398481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.399331 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.399334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.399430 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.400056 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.400071 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.400688 (+ 617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.402848 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.402851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.402853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.403130 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.403481 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.403487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.403491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.423413 (+ 19922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.424223 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.424227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.425025 (+ 798us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.425887 (+ 862us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.426083 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":288,"threads_started":1}
I20260812 11:15:26.431160 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.394275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.0.0.1:33290)
0812 11:15:26.396120 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.396143 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.398815 (+ 2672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.399265 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.399271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.399580 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.399754 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.399766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.400945 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.400951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.402008 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.402024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.402881 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.402885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.403000 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.403727 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.403746 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.423769 (+ 20023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.428490 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.428494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.428497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.428855 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.429496 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.429499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.429501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.429587 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.430112 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.430115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.430238 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.430345 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.430552 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260812 11:15:26.432286 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.394621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.27.109.60:45539)
0812 11:15:26.394727 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.394732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.396241 (+ 1509us) server_negotiation.cc:407] Connection header received
0812 11:15:26.398981 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.398985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.399036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.399111 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.399897 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.400778 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.402188 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.402564 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.402651 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.424023 (+ 21372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.424029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.424036 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.424152 (+ 116us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.427515 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.429270 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.429275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.429277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.429387 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.429675 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.429678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.429680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.429966 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.430074 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.431868 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.432102 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 11:15:26.435137 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.393649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.27.109.61:39379)
0812 11:15:26.393806 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.393811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.394695 (+ 884us) server_negotiation.cc:407] Connection header received
0812 11:15:26.395037 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.395044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.395133 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.395214 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.396698 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.397809 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.403434 (+ 5625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.403952 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.404078 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.425052 (+ 20974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.425060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.425066 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.425118 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.428307 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.433136 (+ 4829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.433141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.433145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.433290 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.433720 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.433724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.433726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.434071 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.434175 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.434590 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.434811 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:15:26.435938 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.393023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.0.0.1:47872)
0812 11:15:26.394521 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.394545 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.394864 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.395449 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.395455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.396283 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.396498 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.396510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.398014 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.398021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.403220 (+ 5199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.403233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.404303 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.404310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.404441 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.405105 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.405128 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.424762 (+ 19634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.432609 (+ 7847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.432614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.432616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.432927 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.433453 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.433457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.433459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.433578 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.434249 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.434252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.434366 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.435454 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.435662 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"spinlock_wait_cycles":8448,"thread_start_us":151,"threads_started":1}
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
I20260812 11:15:26.446412 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.432482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.27.109.60:45539)
0812 11:15:26.433791 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.433796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.434024 (+ 228us) server_negotiation.cc:407] Connection header received
0812 11:15:26.434254 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.434257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.434315 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.434367 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.435086 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.435922 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.437612 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.437953 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.438031 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.442252 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.442256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.442260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.442289 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.444136 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.444819 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.444823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.444825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.444910 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.445299 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.445302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.445304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.445557 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.445631 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.446063 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.446222 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221}
I20260812 11:15:26.447355 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.431977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.0.0.1:33302)
0812 11:15:26.433945 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.433960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.434097 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.434537 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.434539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.434793 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.434942 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.434950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.436458 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.436463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.437375 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.437450 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.438228 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.438255 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.438359 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.439048 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.440262 (+ 1214us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.442043 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.444429 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.444433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.444435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.444698 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.445019 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.445022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.445024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.445174 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.445763 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.445766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.445877 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.446966 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.447161 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1893,"thread_start_us":93,"threads_started":1}
I20260812 11:15:26.448570 4954 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.109.60:45539: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b"
member_type: NON_VOTER
last_known_addr {
host: "127.27.109.60"
port: 45539
}
attrs {
promote: true
}
}
dest_uuid: "ac156fbd8c8f48938db03a94ffeea18e"
cas_config_opid_index: 5
I20260812 11:15:26.449294 4954 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } }
I20260812 11:15:26.452250 6065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:26.453918 6065 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Refusing update from remote peer ac156fbd8c8f48938db03a94ffeea18e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 11:15:26.454615 6143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.457875 6143 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.109.60:45539
I20260812 11:15:26.458431 6143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:26.458873 5875 master.cc:537] Master@127.27.109.60:45539 shutting down...
I20260812 11:15:26.459635 6143 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.459837 6143 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:26.461637 4931 consensus_peers.cc:487] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e -> Peer b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60:45539): Couldn't send request to peer b65a1cd651f84a9d9e0004637f6e5e1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 11:15:26.464625 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.451421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.27.109.61:39379)
0812 11:15:26.451650 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.451658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.454986 (+ 3328us) server_negotiation.cc:407] Connection header received
0812 11:15:26.455331 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.455335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.455408 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.455483 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:26.455920 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.456757 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.462457 (+ 5700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.463318 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.463428 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.463636 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.464261 (+ 625us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.464461 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:15:26.465185 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.451028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.0.0.1:47882)
0812 11:15:26.454892 (+ 3864us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.454910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.455079 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.455665 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.455668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.455687 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:26.455794 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.455800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.457158 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.457164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.462007 (+ 4843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.462021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.463785 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.463789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.463910 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.463977 (+ 67us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.464905 (+ 928us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.465025 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3740,"thread_start_us":78,"threads_started":1}
I20260812 11:15:26.465936 5945 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 LEARNER]: Refusing update from remote peer ac156fbd8c8f48938db03a94ffeea18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 11:15:26.466486 6144 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 11:15:26.468149 5875 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:26.468801 5875 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:26.469060 5875 tablet_replica.cc:330] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: stopping tablet replica
I20260812 11:15:26.468920 5945 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.471256 5991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac156fbd8c8f48938db03a94ffeea18e. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.471408 5991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: LEARNER
I20260812 11:15:26.471491 5991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: true } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.471570 5991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: LEARNER
I20260812 11:15:26.473234 6146 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: attempting to promote NON_VOTER e5eb9c6a7f4644129c9c0524b6bb5d3a to VOTER
I20260812 11:15:26.473877 6146 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } }
I20260812 11:15:26.475593 5945 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 LEARNER]: Refusing update from remote peer ac156fbd8c8f48938db03a94ffeea18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 11:15:26.476567 6143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 11:15:26.476886 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.476221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.0.0.1:33316)
0812 11:15:26.476640 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.476722 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326}
I20260812 11:15:26.479562 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.479244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.0.0.1:33328)
0812 11:15:26.479432 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.479482 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260812 11:15:26.479795 4931 consensus_peers.cc:487] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e -> Peer b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60:45539): Couldn't send request to peer b65a1cd651f84a9d9e0004637f6e5e1b. Status: Network error: Client connection negotiation failed: client connection to 127.27.109.60:45539: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 11:15:26.481057 5875 master.cc:552] Master@127.27.109.60:45539 shutdown complete.
I20260812 11:15:26.481158 5875 master_runner.cc:305] Clearing existing system tablet
I20260812 11:15:26.481263 6144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.481500 5875 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:26.481768 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:26.482117 5945 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.483409 5875 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 11:15:26.483419 6144 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.483536 6144 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:26.483690 6144 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.483772 6144 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:26.484737 6153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:26.484573 6150 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.484830 6150 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:26.485077 5875 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:26.485198 5875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 11:15:26.485239 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
uuid: "b65a1cd651f84a9d9e0004637f6e5e1b"
format_stamp: "Formatted at 2026-08-12 11:15:26 on dist-test-slave-7z32"
server_key: "a7229c70fd33d9c41066c1f562b3673b"
server_key_iv: "39261516b275cd0fad662d3f2c5d02ed"
server_key_version: "encryptionkey@0"
I20260812 11:15:26.487704 5875 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:26.492240 5875 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 11:15:26.492656 5875 log.cc:1196] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 11:15:26.493222 5875 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Deleting consensus metadata
I20260812 11:15:26.493568 5875 master_runner.cc:315] Copying system tablet from 127.27.109.62:34707
I20260812 11:15:26.495677 5875 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.27.109.62:34707
I20260812 11:15:26.508337 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.496869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.27.109.62:34707)
0812 11:15:26.497060 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.497065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.497669 (+ 604us) server_negotiation.cc:407] Connection header received
0812 11:15:26.497860 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.497864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.497924 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.498034 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.498807 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.499790 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.501288 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.501624 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.501759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.503869 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.503874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.503877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.503915 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.505973 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.506749 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.506753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.506755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.506878 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.507267 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.507272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.507275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.507540 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.507618 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.508005 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.508149 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:15:26.509032 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.496653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:34468)
0812 11:15:26.497483 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.497506 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.497722 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.498106 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.498110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.498454 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.498637 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.498646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.499967 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.499973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.501087 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.501105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.501805 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.501810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.501941 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.502759 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.502786 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.503625 (+ 839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.506199 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.506204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.506207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.506628 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.507002 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.507005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.507007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.507097 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.507701 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.507704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.507813 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.508625 (+ 812us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.508821 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":556,"threads_started":1}
I20260812 11:15:26.509701 4985 tablet_copy_service.cc:140] P ac156fbd8c8f48938db03a94ffeea18e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b65a1cd651f84a9d9e0004637f6e5e1b ({username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:34468)
I20260812 11:15:26.509914 4985 tablet_copy_service.cc:161] P ac156fbd8c8f48938db03a94ffeea18e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b65a1cd651f84a9d9e0004637f6e5e1b at {username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:34468: session id = b65a1cd651f84a9d9e0004637f6e5e1b-00000000000000000000000000000000
I20260812 11:15:26.511593 4985 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 11:15:26.513681 5875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 11:15:26.517285 5875 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: tablet copy: Starting download of 0 data blocks...
I20260812 11:15:26.517612 5875 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: tablet copy: Starting download of 1 WAL segments...
I20260812 11:15:26.520469 5875 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 11:15:26.520836 5711 heartbeater.cc:503] Master 127.27.109.62:34707 was elected leader, sending a full tablet report...
I20260812 11:15:26.524420 4985 tablet_copy_service.cc:342] P ac156fbd8c8f48938db03a94ffeea18e: Request end of tablet copy session b65a1cd651f84a9d9e0004637f6e5e1b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:34468
I20260812 11:15:26.524597 4985 tablet_copy_service.cc:433] P ac156fbd8c8f48938db03a94ffeea18e: ending tablet copy session b65a1cd651f84a9d9e0004637f6e5e1b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b65a1cd651f84a9d9e0004637f6e5e1b
I20260812 11:15:26.536000 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:26.538472 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.538739 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.538924 6172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:26.539987 5875 server_base.cc:1034] running on GCE node
I20260812 11:15:26.540262 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:26.540688 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:26.541932 5875 hybrid_clock.cc:648] HybridClock initialized: now 1786533326541901 us; error 63 us; skew 500 ppm
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:26.544482 5875 init.cc:377] Logged in from keytab as oryx/127.27.109.60@KRBTEST.COM (short username oryx)
I20260812 11:15:26.545522 5875 webserver.cc:460] Webserver started at http://127.27.109.60:45601/ using document root <none> and password file <none>
I20260812 11:15:26.545847 5875 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:26.545913 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:26.548336 5875 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:26.549978 6181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:26.550529 5875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260812 11:15:26.550684 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
uuid: "b65a1cd651f84a9d9e0004637f6e5e1b"
format_stamp: "Formatted at 2026-08-12 11:15:26 on dist-test-slave-7z32"
server_key: "a7229c70fd33d9c41066c1f562b3673b"
server_key_iv: "39261516b275cd0fad662d3f2c5d02ed"
server_key_version: "encryptionkey@0"
I20260812 11:15:26.550891 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:26.580569 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:26.583782 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:26.591603 5875 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:45539
I20260812 11:15:26.592686 6260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:45539 every 8 connection(s)
I20260812 11:15:26.592988 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
I20260812 11:15:26.595088 6263 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:15:26.596114 6263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Bootstrap starting.
I20260812 11:15:26.606041 6263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:15:26.606405 6263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Bootstrap complete.
I20260812 11:15:26.607115 6263 raft_consensus.cc:348] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } }
I20260812 11:15:26.607276 6263 raft_consensus.cc:729] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65a1cd651f84a9d9e0004637f6e5e1b, State: Initialized, Role: LEARNER
I20260812 11:15:26.607412 6263 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } }
I20260812 11:15:26.607791 6269 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:26.607918 6269 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: This master's current role is: LEARNER
I20260812 11:15:26.607975 6263 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:26.611128 6282 catalog_manager.cc:1259] Loaded cluster ID: 5561c6c2e0b14a8ca990e91eb3464e52
I20260812 11:15:26.611208 6282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: loading cluster ID for follower catalog manager: success
I20260812 11:15:26.613068 6282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: acquiring CA information for follower catalog manager: success
I20260812 11:15:26.614310 6282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 11:15:26.633481 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:26.633960 6140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:26.634056 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:26.639122 6140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:26.639257 6140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:26.639370 6140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:26.646239 6140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:37737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=33663
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:26.647943 6140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:26.649358 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:26.663002 6293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.663182 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:26.663169 6140 server_base.cc:1034] running on GCE node
W20260812 11:15:26.663456 6292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:26.664278 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:26.665258 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:26.666522 6140 hybrid_clock.cc:648] HybridClock initialized: now 1786533326666467 us; error 72 us; skew 500 ppm
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:26.670776 6140 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260812 11:15:26.672286 6140 webserver.cc:460] Webserver started at http://127.27.109.1:33663/ using document root <none> and password file <none>
I20260812 11:15:26.673077 6140 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:26.673234 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:26.679646 6140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:26.684376 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:26.686360 6140 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:15:26.686594 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "18168ee7cdda412a90889a1be0714fc2"
format_stamp: "Formatted at 2026-08-12 11:15:24 on dist-test-slave-7z32"
server_key: "49c975fac6f234a7cd543ce8936de209"
server_key_iv: "0857b4260537801c499fd56d0e38f757"
server_key_version: "encryptionkey@0"
I20260812 11:15:26.687409 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:26.709491 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:26.714046 6140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:26.714253 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:26.714982 6140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:26.716012 6140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:26.716079 6140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:26.716125 6140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:26.716140 6140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:26.727972 6140 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37737
I20260812 11:15:26.728085 6480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37737 every 8 connection(s)
I20260812 11:15:26.729321 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:26.738317 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 6140
I20260812 11:15:26.738623 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 5291
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:26.754480 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.732071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:41029 (local address 127.27.109.62:34707)
0812 11:15:26.732277 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.732282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.733467 (+ 1185us) server_negotiation.cc:407] Connection header received
0812 11:15:26.735021 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.735028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.735107 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.735201 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.737214 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.738105 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.746947 (+ 8842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.747220 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.747293 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.750485 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.750490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.750492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.750522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.752110 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.752761 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.752764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.752766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.752851 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.753217 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.753220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.753221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.753434 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.753528 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.754189 (+ 661us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.754322 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 11:15:26.756225 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.732598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:41029)
0812 11:15:26.733252 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.733321 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.734701 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.735576 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.735587 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.736203 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.737024 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.737037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.738300 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.738304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.746691 (+ 8387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.746718 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.747535 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.747538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.747748 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.748474 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.748495 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.750250 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.752320 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.752327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.752347 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.752655 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.752950 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.752952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.752954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.753122 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.753605 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.753612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.753937 (+ 325us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.754656 (+ 719us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.755033 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":6925,"spinlock_wait_cycles":15616,"thread_start_us":169,"threads_started":1}
I20260812 11:15:26.760973 6486 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34707
I20260812 11:15:26.761344 6486 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:26.762095 6486 heartbeater.cc:511] Master 127.27.109.62:34707 requested a full tablet report, sending...
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260812 11:15:26.764676 4954 ts_manager.cc:194] Re-registered known tserver with Master: 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737)
I20260812 11:15:26.765616 4954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:41029
I20260812 11:15:26.767696 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.732194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:40423 (local address 127.27.109.61:39379)
0812 11:15:26.732344 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.732349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.734384 (+ 2035us) server_negotiation.cc:407] Connection header received
0812 11:15:26.735061 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.735065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.735124 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.735185 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.737213 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.738111 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.760369 (+ 22258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.760629 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.760762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.763720 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.763724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.763726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.763755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.765521 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.766112 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.766115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.766117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.766196 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.766643 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.766646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.766648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.766916 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.767019 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.767393 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.767529 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:15:26.768399 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.732598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.27.109.1:40423)
0812 11:15:26.734192 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.734206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.734727 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.735576 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.735587 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.736203 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.737024 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.737037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.738300 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.738304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.760180 (+ 21876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.760199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.760789 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.760794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.760900 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.761653 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.761669 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.763522 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.765679 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.765686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.765690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.765971 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.766428 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.766434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.766437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.766535 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.767062 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.767064 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.767197 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.767922 (+ 725us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.768099 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1489,"mutex_wait_us":508,"spinlock_wait_cycles":4864,"thread_start_us":176,"threads_started":1}
I20260812 11:15:26.769562 6487 heartbeater.cc:347] Connected to a master server at 127.27.109.61:39379
I20260812 11:15:26.769676 6487 heartbeater.cc:464] Registering TS with master...
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
I20260812 11:15:26.769945 6487 heartbeater.cc:511] Master 127.27.109.61:39379 requested a full tablet report, sending...
I20260812 11:15:26.770622 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:39669
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:34707
--webserver_port=41491
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:26.771551 5933 ts_manager.cc:194] Registered new tserver with Master: 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737)
I20260812 11:15:26.774437 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.732542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:49907 (local address 127.27.109.60:45539)
0812 11:15:26.732900 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:26.732906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:26.734054 (+ 1148us) server_negotiation.cc:407] Connection header received
0812 11:15:26.735178 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:26.735182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:26.735243 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:26.735358 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:26.737222 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.738076 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.767206 (+ 29130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.767613 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.767700 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.770437 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:26.770444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:26.770447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:26.770479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:26.772506 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.772991 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.772995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.772997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.773089 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:26.773384 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:26.773389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:26.773391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:26.773603 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:26.773719 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:26.774137 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 11:15:26.774268 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
I20260812 11:15:26.774448 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:26.732598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.27.109.1:49907)
0812 11:15:26.733895 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:26.733912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:26.734701 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:26.735579 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:26.735587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:26.736215 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:26.737014 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.737031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.738298 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.738304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.767040 (+ 28736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:26.767052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:26.767907 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:26.767914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:26.768017 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:26.768526 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:26.768542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:26.770198 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:26.772625 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.772629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.772632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.772875 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.773187 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:26.773191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:26.773193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:26.773287 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:26.773707 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:26.773710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:26.773913 (+ 203us) client_negotiation.cc:769] Sending connection context
0812 11:15:26.774054 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 11:15:26.774247 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":27253,"spinlock_wait_cycles":10624,"thread_start_us":170,"threads_started":1}
I20260812 11:15:26.775298 6483 heartbeater.cc:347] Connected to a master server at 127.27.109.60:45539
I20260812 11:15:26.775522 6483 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:26.775789 6483 heartbeater.cc:511] Master 127.27.109.60:45539 requested a full tablet report, sending...
I20260812 11:15:26.777240 6209 ts_manager.cc:194] Registered new tserver with Master: 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737)
W20260812 11:15:26.945693 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:26.946126 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:26.946180 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:26.951581 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:26.951695 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:26.951772 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:26.960281 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:39669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=41491
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:26.962380 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:26.964289 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:26.978899 6512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:26.979475 6502 server_base.cc:1034] running on GCE node
W20260812 11:15:26.978899 6516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:26.978942 6513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:26.980350 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:26.981351 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:26.982729 6502 hybrid_clock.cc:648] HybridClock initialized: now 1786533326982682 us; error 77 us; skew 500 ppm
Aug 12 11:15:26 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:26.986699 6502 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260812 11:15:26.988399 6502 webserver.cc:460] Webserver started at http://127.27.109.2:41491/ using document root <none> and password file <none>
I20260812 11:15:26.989200 6502 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:26.989305 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:26.995337 6502 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:26.998466 6529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:27.000398 6502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:27.000562 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "0812419ccd80493bb22afa94d94683f6"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "ff284e240f432fbca2c517dc91b26b8f"
server_key_iv: "cf5e0ac4b7056f8723eb8a7bf026775c"
server_key_version: "encryptionkey@0"
I20260812 11:15:27.001315 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:27.020887 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:27.025302 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:27.025559 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:27.026346 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:27.027670 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:27.027748 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:27.027796 6502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:27.027904 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:27.041265 6502 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:39669
I20260812 11:15:27.041391 6723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:39669 every 8 connection(s)
I20260812 11:15:27.042634 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:27.044418 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 6502
I20260812 11:15:27.044656 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 5518
I20260812 11:15:27.088762 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:46381
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:34707
--webserver_port=39673
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:15:27 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
Aug 12 11:15:27 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
Aug 12 11:15:27 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:27.099664 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.059385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:38515 (local address 127.27.109.61:39379)
0812 11:15:27.059801 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.059811 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.079319 (+ 19508us) server_negotiation.cc:407] Connection header received
0812 11:15:27.081433 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.081438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.081532 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.081600 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.084564 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.085444 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.087926 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.088326 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.088417 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.093368 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.093376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.093382 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.093421 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.095746 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.096839 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.096846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.096852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.096968 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.097597 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.097602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.097604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.097914 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.098018 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.099180 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.099372 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260812 11:15:27.101958 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.046043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.27.109.2:38515)
0812 11:15:27.079048 (+ 33005us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.079148 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.080929 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.082181 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.082193 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.082980 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:27.084225 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.084258 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.085730 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.085734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.087714 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.087733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.088729 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.088733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.089021 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.089970 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:27.089999 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:27.093033 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:27.095970 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.095982 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.096014 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.096670 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.097104 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.097111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.097116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.097461 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.098107 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:27.098116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:27.098590 (+ 474us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.099981 (+ 1391us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.100520 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32574,"mutex_wait_us":1,"thread_start_us":4599,"threads_started":1}
I20260812 11:15:27.102479 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.078443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:42381 (local address 127.27.109.60:45539)
0812 11:15:27.078646 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.078654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.083117 (+ 4463us) server_negotiation.cc:407] Connection header received
0812 11:15:27.083432 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.083440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.083517 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.083583 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.085018 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.086021 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.092163 (+ 6142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.092587 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.092697 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.097418 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.097424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.097427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.097469 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.100113 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.100817 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.100821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.100824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.100925 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.101276 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.101281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.101283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.101573 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.101675 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.102091 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.102289 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:15:27.103886 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.082013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:44645)
0812 11:15:27.084795 (+ 2782us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.084815 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.086873 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.087462 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.087468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.087773 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:27.087971 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.087984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.091353 (+ 3369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.091355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.092389 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.092399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.093386 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.093392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.094265 (+ 873us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.095116 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:27.095140 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:27.097675 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:27.100628 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.100631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.100634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.100908 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.101416 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.101419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.101421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.101573 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.102954 (+ 1381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:27.102962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:27.103140 (+ 178us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.103335 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.103595 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2681,"spinlock_wait_cycles":6016,"thread_start_us":130,"threads_started":1}
I20260812 11:15:27.103762 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.079727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.27.109.2:42381)
0812 11:15:27.082993 (+ 3266us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.083015 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.083286 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.083801 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.083808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.084162 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:27.084431 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.084447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.086208 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.086211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.089769 (+ 3558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.089800 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.093139 (+ 3339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.093142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.093267 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.094246 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:27.094264 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:27.096403 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:27.100303 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.100309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.100312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.100678 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.101052 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.101055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.101057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.101156 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.101735 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:27.101738 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:27.101865 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.103272 (+ 1407us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.103488 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3065,"mutex_wait_us":1342,"spinlock_wait_cycles":2432,"thread_start_us":162,"threads_started":1}
I20260812 11:15:27.104422 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.082504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:44645 (local address 127.27.109.62:34707)
0812 11:15:27.082671 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.082679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.084901 (+ 2222us) server_negotiation.cc:407] Connection header received
0812 11:15:27.087067 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.087074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.087156 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.087229 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.089989 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.091082 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.092534 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.092913 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.092991 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.097995 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.097999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.098003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.098039 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.100115 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.101048 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.101052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.101054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.101243 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.101753 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.101757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.101758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.102503 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.102617 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.103950 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.104143 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:15:27.104431 6731 heartbeater.cc:347] Connected to a master server at 127.27.109.61:39379
I20260812 11:15:27.104775 6731 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:27.105213 6725 heartbeater.cc:347] Connected to a master server at 127.27.109.60:45539
I20260812 11:15:27.105399 6725 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:27.105647 6731 heartbeater.cc:511] Master 127.27.109.61:39379 requested a full tablet report, sending...
I20260812 11:15:27.105690 6725 heartbeater.cc:511] Master 127.27.109.60:45539 requested a full tablet report, sending...
I20260812 11:15:27.107529 6209 ts_manager.cc:194] Registered new tserver with Master: 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:27.108105 5933 ts_manager.cc:194] Registered new tserver with Master: 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:27.108137 6727 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34707
I20260812 11:15:27.108240 6727 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:27.108520 6727 heartbeater.cc:511] Master 127.27.109.62:34707 requested a full tablet report, sending...
I20260812 11:15:27.109419 4954 ts_manager.cc:194] Re-registered known tserver with Master: 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:27.110430 4954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:44645
I20260812 11:15:27.110740 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.097533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.0.0.1:33332)
0812 11:15:27.105602 (+ 8069us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.105620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.105797 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.106585 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.106590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.106626 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.106791 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.106797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.108049 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.108053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.109341 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.109352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.110276 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.110279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.110373 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.110398 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.110469 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.110574 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7976,"thread_start_us":92,"threads_started":1}
I20260812 11:15:27.110740 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.098507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.27.109.60:45539)
0812 11:15:27.099118 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.099138 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.105748 (+ 6610us) server_negotiation.cc:407] Connection header received
0812 11:15:27.105954 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.105958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.106044 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.106226 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.107004 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.107863 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.109484 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.110129 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.110264 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.110454 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.110494 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.110589 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":123,"threads_started":1}
I20260812 11:15:27.111960 6229 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 11:15:27.112562 6745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac156fbd8c8f48938db03a94ffeea18e. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:27.112743 6745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: This master's current role is: LEARNER
W20260812 11:15:27.240674 6738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:27.241006 6738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:27.241087 6738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:27.245355 6738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:27.245460 6738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:27.245561 6738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:27.251418 6738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:46381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=39673
--tserver_master_addrs=127.27.109.62:34707,127.27.109.61:39379,127.27.109.60:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:27.252815 6738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:27.254057 6738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:27.267174 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:27.267174 6759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:27.267189 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:27.267831 6738 server_base.cc:1034] running on GCE node
I20260812 11:15:27.268545 6738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:27.269507 6738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:27.270788 6738 hybrid_clock.cc:648] HybridClock initialized: now 1786533327270735 us; error 87 us; skew 500 ppm
Aug 12 11:15:27 dist-test-slave-7z32 krb5kdc[4886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533327, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:27.275082 6738 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260812 11:15:27.276468 6738 webserver.cc:460] Webserver started at http://127.27.109.3:39673/ using document root <none> and password file <none>
I20260812 11:15:27.277251 6738 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:27.277398 6738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:27.282979 6738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:15:27.285862 6771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:27.287887 6738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 11:15:27.288107 6738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "81e79624a9d040aa88f9ae538cc70621"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-7z32"
server_key: "6a91e227da3468d946995b1e4bc4ef1d"
server_key_iv: "b9ed85e7b1013a06b8bcb077df796b8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:27.288793 6738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:27.314513 6738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:27.318583 6738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:27.318872 6738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:27.319567 6738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:27.320567 6738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:27.320647 6738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:27.320720 6738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:27.320765 6738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:27.333113 6738 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:46381
I20260812 11:15:27.333180 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:46381 every 8 connection(s)
I20260812 11:15:27.334338 6738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:27.335405 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 6738
Aug 12 11:15:27 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533327, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
Aug 12 11:15:27 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533327, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260812 11:15:27.353523 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.339435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.27.109.60:45539)
0812 11:15:27.341061 (+ 1626us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.341065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.341077 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:15:27.341107 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.341113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.341157 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.341241 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.342177 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.343059 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.346436 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.346756 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.346860 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.349207 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.349213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.349215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.349247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.351124 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.351816 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.351820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.351822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.351915 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.352263 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.352269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.352273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.352564 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.352680 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.353146 (+ 466us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.353301 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1563}
I20260812 11:15:27.355134 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.339664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.27.109.61:39379)
0812 11:15:27.340170 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.340174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.340183 (+ 9us) server_negotiation.cc:407] Connection header received
0812 11:15:27.340229 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.340233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.340290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.340387 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.341579 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.342414 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.343680 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.344009 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.344120 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.346639 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.346643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.346646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.346672 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.350965 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.352030 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.352036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.352038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.352160 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.352521 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.352524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.352527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.352826 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.352985 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.353913 (+ 928us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.354260 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":107,"threads_started":1}
I20260812 11:15:27.355625 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.339037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.27.109.62:34707)
0812 11:15:27.339930 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.339934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.339950 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:15:27.340003 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.340007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.340065 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.340150 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.342375 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.343168 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.344731 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.345084 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.345187 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.347645 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.347651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.347654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.347681 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.353371 (+ 5690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.354010 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.354013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.354015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.354130 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.354458 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.354465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.354469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.354736 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.354929 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.355284 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.355435 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":71,"threads_started":1}
Aug 12 11:15:27 dist-test-slave-7z32 krb5kdc[4886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533327, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:27.359218 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.339327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:47889 (local address 127.27.109.60:45539)
0812 11:15:27.340673 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.340680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.340699 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:27.340752 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.340756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.340813 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.340918 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.343917 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.344621 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.346603 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.346956 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.347043 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.352086 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.352092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.352097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.352124 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.354353 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.357030 (+ 2677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.357034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.357036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.357170 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.357721 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.357724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.357726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.357970 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.358058 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.358912 (+ 854us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.359060 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246}
I20260812 11:15:27.359789 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.339557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:51053 (local address 127.27.109.61:39379)
0812 11:15:27.340029 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.340036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.340054 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:27.340962 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.340966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.341042 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.341147 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.343269 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.344051 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.346068 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.346450 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.346572 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.352961 (+ 6389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.352966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.352969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.353006 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.357418 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.358000 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.358007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.358011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.358122 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.358441 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.358445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.358447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.358683 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.358786 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.359484 (+ 698us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.359628 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
I20260812 11:15:27.360927 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.337360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.27.109.3:47889)
0812 11:15:27.338940 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.339027 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.340400 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.343354 (+ 2954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.343358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.343604 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:27.343756 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.343765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.345409 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.345412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.346477 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.346487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.347209 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.347215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.347337 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.348151 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:27.348178 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:27.351866 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:27.355535 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.355544 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.355566 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.356885 (+ 1319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.357294 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.357300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.357301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.357595 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.358138 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:27.358148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:27.358583 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.359386 (+ 803us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.359761 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"spinlock_wait_cycles":10752,"thread_start_us":769,"threads_started":1}
I20260812 11:15:27.361552 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.337361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.27.109.3:51053)
0812 11:15:27.339983 (+ 2622us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.339998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.340759 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.341583 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.341596 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.342198 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:27.343084 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.343110 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.344222 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.344228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.345865 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.345882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.346799 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.346806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.347159 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.349264 (+ 2105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:27.349282 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:27.352753 (+ 3471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:27.357577 (+ 4824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.357580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.357583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.357856 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.358222 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.358224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.358226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.358333 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.359060 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:27.359063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:27.359151 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.359273 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.360126 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2488,"mutex_wait_us":1146,"thread_start_us":125,"threads_started":1}
I20260812 11:15:27.362766 6965 heartbeater.cc:347] Connected to a master server at 127.27.109.61:39379
I20260812 11:15:27.363140 6965 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:27.363673 6962 heartbeater.cc:347] Connected to a master server at 127.27.109.60:45539
I20260812 11:15:27.363818 6962 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:27.363844 6965 heartbeater.cc:511] Master 127.27.109.61:39379 requested a full tablet report, sending...
I20260812 11:15:27.364078 6962 heartbeater.cc:511] Master 127.27.109.60:45539 requested a full tablet report, sending...
I20260812 11:15:27.359724 4954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:15:27.365331 5933 ts_manager.cc:194] Registered new tserver with Master: 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381)
I20260812 11:15:27.365345 6209 ts_manager.cc:194] Registered new tserver with Master: 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381)
W20260812 11:15:27.365947 4954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:27.367041 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.337360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:41713)
0812 11:15:27.339237 (+ 1877us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.339253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.340846 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.348041 (+ 7195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.348045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.348283 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:27.348454 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.348466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.349958 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.349962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.351039 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.351055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.356167 (+ 5112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.356174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.356306 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.356902 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:27.356918 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:27.359265 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:27.362999 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.363005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.363008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.363402 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.365074 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:27.365079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:27.365081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:27.365218 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:27.366272 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:27.366276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:27.366410 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.366511 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.366780 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1774,"mutex_wait_us":30,"spinlock_wait_cycles":42496,"thread_start_us":192,"threads_started":1}
I20260812 11:15:27.367594 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.338967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:41713 (local address 127.27.109.62:34707)
0812 11:15:27.339806 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.339813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.339834 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:15:27.347645 (+ 7811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.347651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.347738 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.347847 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.348647 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.349606 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.351597 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.355689 (+ 4092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.355822 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.359756 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.359761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.359764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.359794 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.362408 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.363567 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.363571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.363573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.363696 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.365389 (+ 1693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.365393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.365395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.365706 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.366991 (+ 1285us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.367172 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.367395 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746}
I20260812 11:15:27.368424 6963 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34707
I20260812 11:15:27.368584 6963 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:27.368891 6963 heartbeater.cc:511] Master 127.27.109.62:34707 requested a full tablet report, sending...
I20260812 11:15:27.369737 4952 ts_manager.cc:194] Re-registered known tserver with Master: 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381)
I20260812 11:15:27.370136 6152 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: true } } }
I20260812 11:15:27.370296 6152 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:27.370682 4952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:41713
I20260812 11:15:27.375149 6152 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: attempting to promote NON_VOTER b65a1cd651f84a9d9e0004637f6e5e1b to VOTER
I20260812 11:15:27.376072 6152 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:27.377817 5945 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Refusing update from remote peer ac156fbd8c8f48938db03a94ffeea18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 11:15:27.378079 6229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 LEARNER]: Refusing update from remote peer ac156fbd8c8f48938db03a94ffeea18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 11:15:27.378465 6991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 11:15:27.378866 6991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 11:15:27.380455 6991 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.382247 5945 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.382412 6991 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.382520 6991 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:27.382663 6991 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.382754 6991 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:27.382934 6991 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.383026 6991 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:27.382270 6229 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.384613 7004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.385082 7004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:27.385876 6745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac156fbd8c8f48938db03a94ffeea18e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:27.386049 6745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:27.395066 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.388224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:46381 (local address 127.0.0.1:51228)
0812 11:15:27.388610 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.388627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.388751 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.389466 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.389470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.389499 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.389662 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.389670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.391313 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.391317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.392769 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.392789 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.394411 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.394415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.394528 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.394563 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.394665 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.394816 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260812 11:15:27.395810 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.388434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.27.109.3:46381)
0812 11:15:27.388870 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.388892 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.388942 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:15:27.389053 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.389059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.389279 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.389506 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.389801 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.391149 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.392924 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.394242 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.394380 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.394557 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.394699 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.394853 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":167,"threads_started":1}
I20260812 11:15:27.397564 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.387946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37737 (local address 127.0.0.1:51374)
0812 11:15:27.388169 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.388192 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.388298 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.391756 (+ 3458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.391760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.391783 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.391891 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.391897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.393777 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.393780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.395590 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.395598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.397046 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.397051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.397176 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.397210 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.397281 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.397390 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":50}
I20260812 11:15:27.397677 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.388171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.27.109.1:37737)
0812 11:15:27.390413 (+ 2242us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.390424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.390933 (+ 509us) server_negotiation.cc:407] Connection header received
0812 11:15:27.391056 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.391063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.391370 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.392082 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.392336 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.393611 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.395833 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.396771 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.397015 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.397242 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.397339 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.397476 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1861,"thread_start_us":137,"threads_started":1}
I20260812 11:15:27.398751 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.389357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39669 (local address 127.0.0.1:42612)
0812 11:15:27.390548 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.390563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.390675 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.393730 (+ 3055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.393735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.393767 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.393917 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.393923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.395527 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.395530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.397033 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.397043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.398313 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.398317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.398413 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.398437 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.398511 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.398594 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":3,"thread_start_us":159,"threads_started":1}
I20260812 11:15:27.398859 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.389925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.27.109.2:39669)
0812 11:15:27.392870 (+ 2945us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.392885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.392917 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:15:27.393167 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.393176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.393527 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.393741 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.394033 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.395365 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.397204 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.398147 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.398316 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.398493 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.398556 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.398658 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2798,"thread_start_us":145,"threads_started":1}
I20260812 11:15:27.400539 6856 tablet_service.cc:1467] Processing CreateTablet for tablet c11f4e1f92454ec5b70c9828d44f56b0 (DEFAULT_TABLE table=test-table [id=6f2c29389821453894d589afc24b8dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:27.401460 6385 tablet_service.cc:1467] Processing CreateTablet for tablet c11f4e1f92454ec5b70c9828d44f56b0 (DEFAULT_TABLE table=test-table [id=6f2c29389821453894d589afc24b8dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:27.402364 6856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11f4e1f92454ec5b70c9828d44f56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:27.402496 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11f4e1f92454ec5b70c9828d44f56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:27.406694 6606 tablet_service.cc:1467] Processing CreateTablet for tablet c11f4e1f92454ec5b70c9828d44f56b0 (DEFAULT_TABLE table=test-table [id=6f2c29389821453894d589afc24b8dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:27.408313 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11f4e1f92454ec5b70c9828d44f56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:27.414870 7014 tablet_bootstrap.cc:492] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Bootstrap starting.
I20260812 11:15:27.415704 7015 tablet_bootstrap.cc:492] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Bootstrap starting.
I20260812 11:15:27.417404 7016 tablet_bootstrap.cc:492] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Bootstrap starting.
I20260812 11:15:27.418040 7014 tablet_bootstrap.cc:654] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:27.418607 7015 tablet_bootstrap.cc:654] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:27.419293 7014 log.cc:824] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:27.420252 7015 log.cc:824] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:27.420886 7016 tablet_bootstrap.cc:654] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:27.421929 7016 log.cc:824] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:27.423168 7014 tablet_bootstrap.cc:492] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: No bootstrap required, opened a new log
I20260812 11:15:27.423266 7015 tablet_bootstrap.cc:492] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: No bootstrap required, opened a new log
I20260812 11:15:27.423483 7014 ts_tablet_manager.cc:1397] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260812 11:15:27.423529 7015 ts_tablet_manager.cc:1397] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:15:27.424443 7016 tablet_bootstrap.cc:492] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: No bootstrap required, opened a new log
I20260812 11:15:27.424666 7016 ts_tablet_manager.cc:1397] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 11:15:27.427484 7014 raft_consensus.cc:348] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.427879 7014 raft_consensus.cc:374] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:27.427944 7014 raft_consensus.cc:729] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18168ee7cdda412a90889a1be0714fc2, State: Initialized, Role: FOLLOWER
I20260812 11:15:27.428493 7014 consensus_queue.cc:260] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.430064 7014 ts_tablet_manager.cc:1428] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260812 11:15:27.429921 7015 raft_consensus.cc:348] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.429921 7016 raft_consensus.cc:348] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.430271 6486 heartbeater.cc:503] Master 127.27.109.62:34707 was elected leader, sending a full tablet report...
I20260812 11:15:27.430317 7015 raft_consensus.cc:374] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:27.430318 7016 raft_consensus.cc:374] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:27.430379 7015 raft_consensus.cc:729] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e79624a9d040aa88f9ae538cc70621, State: Initialized, Role: FOLLOWER
I20260812 11:15:27.430379 7016 raft_consensus.cc:729] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0812419ccd80493bb22afa94d94683f6, State: Initialized, Role: FOLLOWER
I20260812 11:15:27.431097 7015 consensus_queue.cc:260] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.432515 7015 ts_tablet_manager.cc:1428] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260812 11:15:27.433312 6963 heartbeater.cc:503] Master 127.27.109.62:34707 was elected leader, sending a full tablet report...
I20260812 11:15:27.434927 7016 consensus_queue.cc:260] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.437309 7016 ts_tablet_manager.cc:1428] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260812 11:15:27.437908 6727 heartbeater.cc:503] Master 127.27.109.62:34707 was elected leader, sending a full tablet report...
I20260812 11:15:27.473606 7020 raft_consensus.cc:482] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:27.473956 7020 raft_consensus.cc:504] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.475433 7020 leader_election.cc:288] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381), 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:27.481140 7024 raft_consensus.cc:482] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:27.481516 7024 raft_consensus.cc:504] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.481962 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.476375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:52371 (local address 127.27.109.3:46381)
0812 11:15:27.476521 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.476526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.476944 (+ 418us) server_negotiation.cc:407] Connection header received
0812 11:15:27.477110 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.477113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.477169 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.477266 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.477699 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.478750 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.480354 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.481123 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.481299 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.481376 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.481637 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.481778 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 11:15:27.482059 6488 tablet.cc:2369] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:27.482441 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.476034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:46381 (local address 127.27.109.1:52371)
0812 11:15:27.476807 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.476829 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.476988 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.477312 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.477315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.477337 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.477530 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.477537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.478972 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.478976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.480181 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.480194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.481299 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.481303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.481430 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.481475 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.482166 (+ 691us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.482279 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":49,"thread_start_us":220,"threads_started":1}
I20260812 11:15:27.483569 7024 leader_election.cc:288] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737), 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381)
I20260812 11:15:27.483712 6890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11f4e1f92454ec5b70c9828d44f56b0" candidate_uuid: "18168ee7cdda412a90889a1be0714fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e79624a9d040aa88f9ae538cc70621" is_pre_election: true
I20260812 11:15:27.484153 6890 raft_consensus.cc:2454] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18168ee7cdda412a90889a1be0714fc2 in term 0.
I20260812 11:15:27.484465 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.476034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39669 (local address 127.27.109.1:51361)
0812 11:15:27.476551 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.476571 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.476742 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.477404 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.477410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.477434 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.477590 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.477598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.479000 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.479004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.480390 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.480405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.483805 (+ 3400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.483812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.483966 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.484006 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.484112 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.484254 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":55,"thread_start_us":258,"threads_started":1}
I20260812 11:15:27.485039 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.476182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:51361 (local address 127.27.109.2:39669)
0812 11:15:27.476642 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.476648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.476666 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:27.477139 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.477142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.477189 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.477266 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.477812 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.478789 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.480589 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.481236 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.481570 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.481633 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.484764 (+ 3131us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.484897 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260812 11:15:27.485119 6311 leader_election.cc:302] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18168ee7cdda412a90889a1be0714fc2, 81e79624a9d040aa88f9ae538cc70621; no voters:
I20260812 11:15:27.485697 7020 raft_consensus.cc:2781] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:27.485730 6642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11f4e1f92454ec5b70c9828d44f56b0" candidate_uuid: "18168ee7cdda412a90889a1be0714fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0812419ccd80493bb22afa94d94683f6" is_pre_election: true
I20260812 11:15:27.485806 7020 raft_consensus.cc:482] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:27.485877 7020 raft_consensus.cc:3037] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:27.486009 6642 raft_consensus.cc:2454] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18168ee7cdda412a90889a1be0714fc2 in term 0.
I20260812 11:15:27.487485 7020 raft_consensus.cc:504] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.487932 7020 leader_election.cc:288] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [CANDIDATE]: Term 1 election: Requested vote from peers 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381), 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:27.488493 6642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11f4e1f92454ec5b70c9828d44f56b0" candidate_uuid: "18168ee7cdda412a90889a1be0714fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0812419ccd80493bb22afa94d94683f6"
I20260812 11:15:27.488684 6890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11f4e1f92454ec5b70c9828d44f56b0" candidate_uuid: "18168ee7cdda412a90889a1be0714fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e79624a9d040aa88f9ae538cc70621"
I20260812 11:15:27.488704 6642 raft_consensus.cc:3037] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:27.488862 6890 raft_consensus.cc:3037] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:27.489425 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.483881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:46381 (local address 127.27.109.2:46687)
0812 11:15:27.484366 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.484380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.484444 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.485068 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.485071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.485088 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.485246 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.485255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.486554 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.486558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.487820 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.487833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.488813 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.488818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.488955 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.489003 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.489084 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.489210 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":62}
I20260812 11:15:27.489477 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.483769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37737 (local address 127.27.109.2:43111)
0812 11:15:27.484099 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:27.484115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:27.484211 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:27.485472 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:27.485477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:27.485502 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:27.485597 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.485603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.486606 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.486609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.487941 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:27.487956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.488949 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.488952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:27.489066 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.489098 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:15:27.489183 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 11:15:27.489313 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":1}
I20260812 11:15:27.489992 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.484632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:43111 (local address 127.27.109.1:37737)
0812 11:15:27.484756 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.484761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.484773 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:15:27.484819 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.484822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.484871 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.484951 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.485721 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.486461 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.488107 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.488689 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.488810 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.488872 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.489779 (+ 907us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.489873 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:15:27.490170 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.484184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:46687 (local address 127.27.109.3:46381)
0812 11:15:27.484502 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.484509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.484529 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:15:27.484589 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.484592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.484661 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.484754 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:27.485459 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.486412 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.487989 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.488642 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.489799 (+ 1157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.489868 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.489912 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.490009 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260812 11:15:27.490636 6890 raft_consensus.cc:2454] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18168ee7cdda412a90889a1be0714fc2 in term 1.
I20260812 11:15:27.490617 6415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11f4e1f92454ec5b70c9828d44f56b0" candidate_uuid: "0812419ccd80493bb22afa94d94683f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18168ee7cdda412a90889a1be0714fc2" is_pre_election: true
I20260812 11:15:27.490665 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11f4e1f92454ec5b70c9828d44f56b0" candidate_uuid: "0812419ccd80493bb22afa94d94683f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e79624a9d040aa88f9ae538cc70621" is_pre_election: true
I20260812 11:15:27.490931 6642 raft_consensus.cc:2454] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18168ee7cdda412a90889a1be0714fc2 in term 1.
I20260812 11:15:27.490957 6415 raft_consensus.cc:2379] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0812419ccd80493bb22afa94d94683f6 in current term 1: Already voted for candidate 18168ee7cdda412a90889a1be0714fc2 in this term.
I20260812 11:15:27.491003 6888 raft_consensus.cc:2379] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0812419ccd80493bb22afa94d94683f6 in current term 1: Already voted for candidate 18168ee7cdda412a90889a1be0714fc2 in this term.
I20260812 11:15:27.491421 6309 leader_election.cc:302] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0812419ccd80493bb22afa94d94683f6, 18168ee7cdda412a90889a1be0714fc2; no voters:
I20260812 11:15:27.491695 7020 raft_consensus.cc:2781] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:27.491858 6536 leader_election.cc:302] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0812419ccd80493bb22afa94d94683f6; no voters: 18168ee7cdda412a90889a1be0714fc2, 81e79624a9d040aa88f9ae538cc70621
I20260812 11:15:27.491979 7020 raft_consensus.cc:686] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 1 LEADER]: Becoming Leader. State: Replica: 18168ee7cdda412a90889a1be0714fc2, State: Running, Role: LEADER
I20260812 11:15:27.492229 7024 raft_consensus.cc:2726] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 11:15:27.492426 7020 consensus_queue.cc:237] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } }
I20260812 11:15:27.497391 4952 catalog_manager.cc:5495] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "18168ee7cdda412a90889a1be0714fc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:27.534998 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.530582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.27.109.1:37737)
0812 11:15:27.530719 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.530723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.530740 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:15:27.531003 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.531007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.531076 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.531202 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:27.531492 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.532013 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.533063 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.533340 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.533448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.533842 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:27.534416 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:27.534532 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.534732 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.534796 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:15:27.543527 6888 raft_consensus.cc:1261] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Refusing update from remote peer 18168ee7cdda412a90889a1be0714fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:27.543644 6642 raft_consensus.cc:1261] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Refusing update from remote peer 18168ee7cdda412a90889a1be0714fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:27.544658 7020 consensus_queue.cc:1035] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:27.545393 7027 consensus_queue.cc:1035] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:27.557581 7030 mvcc.cc:205] Tried to move back new op lower bound from 7317640509607784448 to 7317640509413945344. Current Snapshot: MvccSnapshot[applied={T|T < 7317640509607784448}]
I20260812 11:15:27.562261 7033 mvcc.cc:205] Tried to move back new op lower bound from 7317640509607784448 to 7317640509413945344. Current Snapshot: MvccSnapshot[applied={T|T < 7317640509607784448}]
I20260812 11:15:27.571274 4954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:27.571557 4954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:27.578181 4954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac156fbd8c8f48938db03a94ffeea18e: Sending DeleteTablet for 3 replicas of tablet c11f4e1f92454ec5b70c9828d44f56b0
I20260812 11:15:27.579322 6856 tablet_service.cc:1514] Processing DeleteTablet for tablet c11f4e1f92454ec5b70c9828d44f56b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260812 11:15:27.579417 6606 tablet_service.cc:1514] Processing DeleteTablet for tablet c11f4e1f92454ec5b70c9828d44f56b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42612
I20260812 11:15:27.580049 7050 tablet_replica.cc:330] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: stopping tablet replica
I20260812 11:15:27.580113 6385 tablet_service.cc:1514] Processing DeleteTablet for tablet c11f4e1f92454ec5b70c9828d44f56b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:51374
I20260812 11:15:27.580508 7050 raft_consensus.cc:2229] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:27.580704 7053 tablet_replica.cc:330] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: stopping tablet replica
I20260812 11:15:27.581036 7053 raft_consensus.cc:2229] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:27.581013 7050 raft_consensus.cc:2258] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:27.581466 7053 raft_consensus.cc:2258] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:27.583603 7050 ts_tablet_manager.cc:1905] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:27.583603 7053 ts_tablet_manager.cc:1905] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:27.584470 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 4903
I20260812 11:15:27.611760 7050 ts_tablet_manager.cc:1918] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:27.611768 7053 ts_tablet_manager.cc:1918] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:27.611930 7050 log.cc:1196] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/c11f4e1f92454ec5b70c9828d44f56b0
I20260812 11:15:27.611929 7053 log.cc:1196] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/c11f4e1f92454ec5b70c9828d44f56b0
I20260812 11:15:27.612353 7053 ts_tablet_manager.cc:1939] T c11f4e1f92454ec5b70c9828d44f56b0 P 18168ee7cdda412a90889a1be0714fc2: Deleting consensus metadata
I20260812 11:15:27.612362 7050 ts_tablet_manager.cc:1939] T c11f4e1f92454ec5b70c9828d44f56b0 P 81e79624a9d040aa88f9ae538cc70621: Deleting consensus metadata
I20260812 11:15:27.624167 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.582616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.27.109.61:39379)
0812 11:15:27.582809 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.582815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.614173 (+ 31358us) server_negotiation.cc:407] Connection header received
0812 11:15:27.614359 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.614366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.614436 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.614519 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.615626 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.616480 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.617710 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.618065 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.618138 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.620289 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.620293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.620296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.620326 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.622130 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.622680 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.622684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.622686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.622773 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.623116 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.623120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.623121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.623362 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.623486 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.623844 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.623990 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:15:27.629971 7052 tablet_replica.cc:330] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: stopping tablet replica
I20260812 11:15:27.630566 7052 raft_consensus.cc:2229] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 11:15:27.632282 6778 connection.cc:535] server connection from 127.0.0.1:51228 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:15:27.632736 7052 raft_consensus.cc:2258] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 11:15:27.632824 6309 connection.cc:535] server connection from 127.0.0.1:51374 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:15:27.634450 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.582764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.27.109.60:45539)
0812 11:15:27.582954 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.582959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.624831 (+ 41872us) server_negotiation.cc:407] Connection header received
0812 11:15:27.625040 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.625044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.625103 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.625162 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.626074 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.626951 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.628127 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.628514 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.628591 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.630444 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.630448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.630451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.630481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.632247 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.632754 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.632757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.632759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.632866 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.633405 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.633409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.633411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.633636 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.633750 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.634114 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.634267 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 11:15:27.636404 7052 ts_tablet_manager.cc:1905] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:27.640527 7052 ts_tablet_manager.cc:1918] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:27.640823 7052 log.cc:1196] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/c11f4e1f92454ec5b70c9828d44f56b0
I20260812 11:15:27.641425 7052 ts_tablet_manager.cc:1939] T c11f4e1f92454ec5b70c9828d44f56b0 P 0812419ccd80493bb22afa94d94683f6: Deleting consensus metadata
W20260812 11:15:27.643291 6536 connection.cc:441] server connection from 127.0.0.1:42612 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42612 (request call id 1) could send its response
I20260812 11:15:27.652416 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.638098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.27.109.60:45539)
0812 11:15:27.638215 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.638219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.639754 (+ 1535us) server_negotiation.cc:407] Connection header received
0812 11:15:27.640116 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.640120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.640188 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.640253 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.642576 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.643395 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.644574 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.644867 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.644989 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.647509 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.647514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.647517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.647555 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.649904 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.650645 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.650650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.650654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.650744 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.651205 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.651209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.651210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.651537 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.651690 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.652077 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.652271 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:15:27.652416 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.638225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.27.109.61:39379)
0812 11:15:27.638422 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.638427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.639772 (+ 1345us) server_negotiation.cc:407] Connection header received
0812 11:15:27.639990 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.639994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.640053 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.640113 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.641273 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.642104 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.643555 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:27.643925 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:27.644004 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:27.647620 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.647627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.647634 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:27.647672 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.649936 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.650574 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.650578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.650582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.650693 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:27.651320 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:27.651324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:27.651325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:27.651555 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.651686 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.652042 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.652214 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:15:28.561769 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:28.561078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:60903)
0812 11:15:28.561522 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:28.561634 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":98,"threads_started":1}
W20260812 11:15:28.562296 6486 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:34707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:28.562443 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:28.561550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:34589)
0812 11:15:28.562205 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:28.562310 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":108,"threads_started":1}
W20260812 11:15:28.563026 6727 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:34707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:28.564841 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:28.564290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:59769)
0812 11:15:28.564583 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:28.564714 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
W20260812 11:15:28.565404 6963 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:34707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:29.079039 7070 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac156fbd8c8f48938db03a94ffeea18e)
I20260812 11:15:29.079124 7069 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac156fbd8c8f48938db03a94ffeea18e)
I20260812 11:15:29.079433 7070 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:29.079572 7069 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:29.081740 7070 leader_election.cc:288] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707), e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61:39379)
I20260812 11:15:29.082265 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.081241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49426)
0812 11:15:29.082044 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.082147 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":160,"threads_started":1}
W20260812 11:15:29.082506 5912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:29.082631 7069 leader_election.cc:288] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707), b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60:45539)
I20260812 11:15:29.083704 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.082759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49422)
0812 11:15:29.083472 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.083559 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":95,"threads_started":1}
W20260812 11:15:29.084029 6190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:29.085129 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.084838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49452)
0812 11:15:29.084983 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.085043 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260812 11:15:29.085292 6190 leader_election.cc:334] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707): Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:29.085916 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.084384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49438)
0812 11:15:29.085724 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.085820 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260812 11:15:29.086045 5912 leader_election.cc:334] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707): Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:29.088697 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.082876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.27.109.60:45539)
0812 11:15:29.083560 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:29.083566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:29.083584 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:29.083635 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:29.083638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:29.083686 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:29.083797 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:29.084584 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.085509 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.087190 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.087998 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.088130 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:29.088208 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:29.088463 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 11:15:29.088554 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":72,"threads_started":1}
I20260812 11:15:29.088743 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.082423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:45539 (local address 127.0.0.1:40710)
0812 11:15:29.083057 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.083074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:29.083237 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:29.084278 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:29.084282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:29.084304 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:29.084440 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:29.084447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.085775 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.085781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:29.087004 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:29.087020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.088168 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.088174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:29.088284 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:29.088326 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:15:29.088419 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 11:15:29.088523 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":79,"threads_started":1}
I20260812 11:15:29.088840 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.081310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39379 (local address 127.0.0.1:40074)
0812 11:15:29.083120 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.083139 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:29.083280 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:29.084921 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:29.084925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:29.084948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:29.085062 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:29.085067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.086159 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.086163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:29.087311 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:29.087326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.088335 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.088337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:29.088461 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:29.088485 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:15:29.088570 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 11:15:29.088655 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"thread_start_us":105,"threads_started":1}
I20260812 11:15:29.089429 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.083416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.27.109.61:39379)
0812 11:15:29.084678 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:29.084687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:29.084698 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:15:29.084732 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:29.084735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:29.084777 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:29.085155 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:29.085237 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.086005 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.087449 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.088205 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.088340 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:29.088421 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:29.088621 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 11:15:29.088698 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":108,"threads_started":1}
I20260812 11:15:29.089445 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" is_pre_election: true
I20260812 11:15:29.089752 6229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5eb9c6a7f4644129c9c0524b6bb5d3a in term 1.
I20260812 11:15:29.089979 5945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" is_pre_election: true
I20260812 11:15:29.090293 5945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65a1cd651f84a9d9e0004637f6e5e1b in term 1.
I20260812 11:15:29.090297 5911 leader_election.cc:302] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b65a1cd651f84a9d9e0004637f6e5e1b, e5eb9c6a7f4644129c9c0524b6bb5d3a; no voters: ac156fbd8c8f48938db03a94ffeea18e
I20260812 11:15:29.090628 7069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 11:15:29.090698 7069 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac156fbd8c8f48938db03a94ffeea18e)
I20260812 11:15:29.090752 7069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:15:29.090727 6185 leader_election.cc:302] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b65a1cd651f84a9d9e0004637f6e5e1b, e5eb9c6a7f4644129c9c0524b6bb5d3a; no voters: ac156fbd8c8f48938db03a94ffeea18e
I20260812 11:15:29.091037 7070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 11:15:29.091127 7070 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac156fbd8c8f48938db03a94ffeea18e)
I20260812 11:15:29.091197 7070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:15:29.092962 7069 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:29.093027 7070 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:29.093524 7070 leader_election.cc:288] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 2 election: Requested vote from peers ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707), e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61:39379)
I20260812 11:15:29.093524 7069 leader_election.cc:288] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [CANDIDATE]: Term 2 election: Requested vote from peers ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707), b65a1cd651f84a9d9e0004637f6e5e1b (127.27.109.60:45539)
I20260812 11:15:29.094276 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.093857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49476)
0812 11:15:29.094080 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.094163 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 11:15:29.095278 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.094496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49488)
0812 11:15:29.095136 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.095201 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567}
I20260812 11:15:29.095299 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.094962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49500)
0812 11:15:29.095138 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.095209 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260812 11:15:29.095425 6190 leader_election.cc:334] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707): Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:29.095793 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b"
I20260812 11:15:29.096045 6229 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e5eb9c6a7f4644129c9c0524b6bb5d3a in current term 2: Already voted for candidate b65a1cd651f84a9d9e0004637f6e5e1b in this term.
I20260812 11:15:29.096555 5945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
I20260812 11:15:29.096668 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.096400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49504)
0812 11:15:29.096518 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.096582 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 11:15:29.096748 5945 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b65a1cd651f84a9d9e0004637f6e5e1b in current term 2: Already voted for candidate e5eb9c6a7f4644129c9c0524b6bb5d3a in this term.
W20260812 11:15:29.096935 5912 leader_election.cc:334] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707): Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:29.097018 5912 leader_election.cc:302] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5eb9c6a7f4644129c9c0524b6bb5d3a; no voters: ac156fbd8c8f48938db03a94ffeea18e, b65a1cd651f84a9d9e0004637f6e5e1b
I20260812 11:15:29.097235 7069 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 11:15:29.097216 6185 leader_election.cc:302] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b65a1cd651f84a9d9e0004637f6e5e1b; no voters: ac156fbd8c8f48938db03a94ffeea18e, e5eb9c6a7f4644129c9c0524b6bb5d3a
I20260812 11:15:29.097536 7070 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 11:15:29.563812 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.563142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:36177)
0812 11:15:29.563601 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.563679 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":147,"threads_started":1}
I20260812 11:15:29.565369 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.564142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:55913)
0812 11:15:29.565101 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.565225 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":204,"threads_started":1}
I20260812 11:15:29.566699 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.566181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:52691)
0812 11:15:29.566473 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:29.566552 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20260812 11:15:30.566022 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.565100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:36097)
0812 11:15:30.565772 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.565884 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":135,"threads_started":1}
I20260812 11:15:30.566959 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.566310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:35457)
0812 11:15:30.566684 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.566775 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260812 11:15:30.568128 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.567707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:57537)
0812 11:15:30.567974 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.568032 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260812 11:15:30.707690 7094 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:30.707924 7094 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:30.708640 7094 leader_election.cc:288] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707), e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61:39379)
I20260812 11:15:30.709517 5945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" is_pre_election: true
I20260812 11:15:30.709710 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.709127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49732)
0812 11:15:30.709487 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.709564 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":159,"threads_started":1}
I20260812 11:15:30.709824 5945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65a1cd651f84a9d9e0004637f6e5e1b in term 2.
I20260812 11:15:30.710407 6185 leader_election.cc:302] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65a1cd651f84a9d9e0004637f6e5e1b, e5eb9c6a7f4644129c9c0524b6bb5d3a; no voters:
I20260812 11:15:30.710893 7094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 11:15:30.711067 7094 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:30.711141 7094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:15:30.711187 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.710745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49740)
0812 11:15:30.710906 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.711099 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:15:30.711355 6190 leader_election.cc:334] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707): Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:30.712777 7094 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:30.713241 7094 leader_election.cc:288] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 3 election: Requested vote from peers ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707), e5eb9c6a7f4644129c9c0524b6bb5d3a (127.27.109.61:39379)
I20260812 11:15:30.713861 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.713528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49744)
0812 11:15:30.713708 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.713781 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 11:15:30.714021 5945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a"
I20260812 11:15:30.714283 5945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:15:30.714723 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.714476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49760)
0812 11:15:30.714608 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.714662 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 11:15:30.714898 6190 leader_election.cc:334] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707): Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
I20260812 11:15:30.715909 5945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65a1cd651f84a9d9e0004637f6e5e1b in term 3.
I20260812 11:15:30.716401 6185 leader_election.cc:302] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b65a1cd651f84a9d9e0004637f6e5e1b, e5eb9c6a7f4644129c9c0524b6bb5d3a; no voters: ac156fbd8c8f48938db03a94ffeea18e
I20260812 11:15:30.716746 7094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 11:15:30.717052 7094 raft_consensus.cc:686] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [term 3 LEADER]: Becoming Leader. State: Replica: b65a1cd651f84a9d9e0004637f6e5e1b, State: Running, Role: LEADER
I20260812 11:15:30.717382 7094 consensus_queue.cc:237] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } }
I20260812 11:15:30.719336 7097 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b65a1cd651f84a9d9e0004637f6e5e1b. Latest consensus state: current_term: 3 leader_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:30.719523 7097 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:30.720027 7102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:30.722709 7102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:30.723277 7102 catalog_manager.cc:1259] Loaded cluster ID: 5561c6c2e0b14a8ca990e91eb3464e52
I20260812 11:15:30.723337 7102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:30.723999 7102 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:30.724462 7102 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Loaded TSK: 0
I20260812 11:15:30.725272 7102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:30.774566 6209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:30.778331 6209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:30.782888 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.782449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49776)
0812 11:15:30.782639 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.782738 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 11:15:30.783015 5945 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [term 3 FOLLOWER]: Refusing update from remote peer b65a1cd651f84a9d9e0004637f6e5e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 11:15:30.783776 7097 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 11:15:30.783972 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.783730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.0.0.1:49780)
0812 11:15:30.783829 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.783893 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 11:15:30.784161 6190 consensus_peers.cc:487] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b -> Peer ac156fbd8c8f48938db03a94ffeea18e (127.27.109.62:34707): Couldn't send request to peer ac156fbd8c8f48938db03a94ffeea18e. Status: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 11:15:30.785878 7106 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b65a1cd651f84a9d9e0004637f6e5e1b. Latest consensus state: current_term: 3 leader_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:30.786019 7106 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:30.788013 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:30.788223 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:30.788560 7106 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac156fbd8c8f48938db03a94ffeea18e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34707 } } peers { permanent_uuid: "e5eb9c6a7f4644129c9c0524b6bb5d3a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39379 } attrs { promote: false } } peers { permanent_uuid: "b65a1cd651f84a9d9e0004637f6e5e1b" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 45539 } attrs { promote: false } } }
I20260812 11:15:30.788745 7106 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5eb9c6a7f4644129c9c0524b6bb5d3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:30.789043 7107 mvcc.cc:205] Tried to move back new op lower bound from 7317640522879012864 to 7317640522621325312. Current Snapshot: MvccSnapshot[applied={T|T < 7317640522879012864}]
I20260812 11:15:30.805704 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.799705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37737 (local address 127.0.0.1:60700)
0812 11:15:30.799986 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.800013 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:30.800125 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:30.801316 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:30.801320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:30.801342 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:30.801683 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.801695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.802993 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.802997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.804285 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.804298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.805220 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.805224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.805343 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.805375 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:15:30.805438 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 11:15:30.805532 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 11:15:30.806074 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.800366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:46381 (local address 127.0.0.1:39070)
0812 11:15:30.800748 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.800768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:30.800906 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:30.801945 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:30.801949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:30.801971 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:30.802052 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.802058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.803147 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.803151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.804295 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.804306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.805484 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.805488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.805616 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.805650 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:15:30.805740 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:15:30.805854 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":49,"thread_start_us":195,"threads_started":1}
I20260812 11:15:30.806402 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.800728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39669 (local address 127.0.0.1:38412)
0812 11:15:30.801689 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.801705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:30.801825 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:30.802191 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:30.802195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:30.802213 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:30.802354 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.802360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.803475 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.803479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.804736 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.804745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.805910 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.805915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.806022 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.806058 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:15:30.806174 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:15:30.806259 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":32,"thread_start_us":169,"threads_started":1}
I20260812 11:15:30.806499 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.800307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.27.109.1:37737)
0812 11:15:30.800625 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:30.800632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:30.800996 (+ 364us) server_negotiation.cc:407] Connection header received
0812 11:15:30.801069 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:30.801080 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:30.801161 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:30.801830 (+ 669us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:30.801946 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.802854 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.804471 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.805076 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.806147 (+ 1071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.806219 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:30.806255 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 11:15:30.806330 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":131,"threads_started":1}
I20260812 11:15:30.806612 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.800933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.27.109.3:46381)
0812 11:15:30.801316 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:30.801321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:30.801337 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:15:30.801578 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:30.801582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:30.801644 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:30.801764 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:30.802197 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.803012 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.804470 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.805191 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.805329 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.805419 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:30.806270 (+ 851us) server_negotiation.cc:299] Negotiation successful
0812 11:15:30.806402 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
I20260812 11:15:30.806939 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.800848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.27.109.2:39669)
0812 11:15:30.801528 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:30.801534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:30.801927 (+ 393us) server_negotiation.cc:407] Connection header received
0812 11:15:30.802002 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:30.802008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:30.802074 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:30.802181 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:30.802483 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.803323 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.804920 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.805628 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.805752 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.805835 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:30.806668 (+ 833us) server_negotiation.cc:299] Negotiation successful
0812 11:15:30.806759 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":77,"threads_started":1}
I20260812 11:15:30.807129 6385 tablet_service.cc:1467] Processing CreateTablet for tablet c616320e73644258b82416d89f26ddef (DEFAULT_TABLE table=test-table [id=dc98892f703f4315ae8795d507478ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:30.807559 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c616320e73644258b82416d89f26ddef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:30.807592 6856 tablet_service.cc:1467] Processing CreateTablet for tablet c616320e73644258b82416d89f26ddef (DEFAULT_TABLE table=test-table [id=dc98892f703f4315ae8795d507478ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:30.807626 6606 tablet_service.cc:1467] Processing CreateTablet for tablet c616320e73644258b82416d89f26ddef (DEFAULT_TABLE table=test-table [id=dc98892f703f4315ae8795d507478ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:30.807971 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c616320e73644258b82416d89f26ddef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:30.807971 6856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c616320e73644258b82416d89f26ddef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:30.810111 7121 tablet_bootstrap.cc:492] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2: Bootstrap starting.
I20260812 11:15:30.811378 7121 tablet_bootstrap.cc:654] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:30.811519 7122 tablet_bootstrap.cc:492] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Bootstrap starting.
I20260812 11:15:30.812561 7122 tablet_bootstrap.cc:654] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:30.813133 7121 tablet_bootstrap.cc:492] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2: No bootstrap required, opened a new log
I20260812 11:15:30.813277 7121 ts_tablet_manager.cc:1397] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:30.813467 7123 tablet_bootstrap.cc:492] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Bootstrap starting.
I20260812 11:15:30.813743 7121 raft_consensus.cc:348] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.813885 7121 raft_consensus.cc:374] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:30.813931 7121 raft_consensus.cc:729] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18168ee7cdda412a90889a1be0714fc2, State: Initialized, Role: FOLLOWER
I20260812 11:15:30.814152 7121 consensus_queue.cc:260] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.814674 7123 tablet_bootstrap.cc:654] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:30.814755 7122 tablet_bootstrap.cc:492] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: No bootstrap required, opened a new log
I20260812 11:15:30.814932 7122 ts_tablet_manager.cc:1397] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:30.815758 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.815386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:49101)
0812 11:15:30.815541 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.815612 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 11:15:30.815645 7122 raft_consensus.cc:348] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.815814 7122 raft_consensus.cc:374] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:30.815852 7122 raft_consensus.cc:729] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0812419ccd80493bb22afa94d94683f6, State: Initialized, Role: FOLLOWER
I20260812 11:15:30.816020 7122 consensus_queue.cc:260] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.815642 7121 ts_tablet_manager.cc:1428] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:15:30.816550 7122 ts_tablet_manager.cc:1428] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:30.817028 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.816743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:56479)
0812 11:15:30.816883 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.816947 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 11:15:30.818183 7123 tablet_bootstrap.cc:492] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: No bootstrap required, opened a new log
I20260812 11:15:30.818329 7123 ts_tablet_manager.cc:1397] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 11:15:30.818730 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.817429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:48363)
0812 11:15:30.818540 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.818626 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1002}
I20260812 11:15:30.818818 7123 raft_consensus.cc:348] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.819023 7123 raft_consensus.cc:374] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:30.819068 7123 raft_consensus.cc:729] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e79624a9d040aa88f9ae538cc70621, State: Initialized, Role: FOLLOWER
I20260812 11:15:30.819238 7123 consensus_queue.cc:260] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.819674 7123 ts_tablet_manager.cc:1428] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 11:15:30.819733 6725 heartbeater.cc:503] Master 127.27.109.60:45539 was elected leader, sending a full tablet report...
I20260812 11:15:30.820462 6483 heartbeater.cc:503] Master 127.27.109.60:45539 was elected leader, sending a full tablet report...
I20260812 11:15:30.820510 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.820205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:41893)
0812 11:15:30.820343 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.820401 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 11:15:30.821504 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.821095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:44423)
0812 11:15:30.821341 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.821414 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260812 11:15:30.823086 6962 heartbeater.cc:503] Master 127.27.109.60:45539 was elected leader, sending a full tablet report...
I20260812 11:15:30.823567 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.822743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:38783)
0812 11:15:30.822965 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.823039 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260812 11:15:30.832983 6733 tablet.cc:2369] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:30.836576 7131 raft_consensus.cc:482] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:30.836793 7131 raft_consensus.cc:504] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.838111 7131 leader_election.cc:288] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737), 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:30.844722 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.838566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39669 (local address 127.27.109.3:39243)
0812 11:15:30.838917 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.838938 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:30.839111 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:30.839556 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:30.839559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:30.839581 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:30.839887 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.839896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.841068 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.841072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.843056 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.843075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.844225 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.844228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.844345 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.844374 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:15:30.844450 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 11:15:30.844586 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":239}
I20260812 11:15:30.844753 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.838706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39243 (local address 127.27.109.2:39669)
0812 11:15:30.838879 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:30.838884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:30.839051 (+ 167us) server_negotiation.cc:407] Connection header received
0812 11:15:30.839284 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:30.839288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:30.839354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:30.839501 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:30.840051 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.840879 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.843285 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.844038 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.844199 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.844275 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:30.844495 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 11:15:30.844600 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:15:30.845436 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.838622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37737 (local address 127.27.109.3:48435)
0812 11:15:30.839156 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.839173 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:30.839285 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:30.839556 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:30.839613 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:30.839630 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:30.839889 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.839896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.841058 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.841063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.843056 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:30.843075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.844956 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.844960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:30.845079 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.845104 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:15:30.845195 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:15:30.845282 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":301,"thread_start_us":224,"threads_started":1}
I20260812 11:15:30.845383 6642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616320e73644258b82416d89f26ddef" candidate_uuid: "81e79624a9d040aa88f9ae538cc70621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0812419ccd80493bb22afa94d94683f6" is_pre_election: true
I20260812 11:15:30.845491 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.838934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48435 (local address 127.27.109.1:37737)
0812 11:15:30.839072 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:30.839079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:30.839272 (+ 193us) server_negotiation.cc:407] Connection header received
0812 11:15:30.839366 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:30.839369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:30.839425 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:30.839514 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:30.840051 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.840908 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.843286 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.844038 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.844199 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.844275 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:30.845249 (+ 974us) server_negotiation.cc:299] Negotiation successful
0812 11:15:30.845374 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:15:30.845594 6642 raft_consensus.cc:2454] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e79624a9d040aa88f9ae538cc70621 in term 0.
I20260812 11:15:30.846113 6415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616320e73644258b82416d89f26ddef" candidate_uuid: "81e79624a9d040aa88f9ae538cc70621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18168ee7cdda412a90889a1be0714fc2" is_pre_election: true
I20260812 11:15:30.846104 6773 leader_election.cc:302] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0812419ccd80493bb22afa94d94683f6, 81e79624a9d040aa88f9ae538cc70621; no voters:
I20260812 11:15:30.846351 6415 raft_consensus.cc:2454] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e79624a9d040aa88f9ae538cc70621 in term 0.
I20260812 11:15:30.846439 7131 raft_consensus.cc:2781] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:30.846527 7131 raft_consensus.cc:482] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:30.846580 7131 raft_consensus.cc:3037] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:30.847832 7131 raft_consensus.cc:504] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.848237 7131 leader_election.cc:288] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [CANDIDATE]: Term 1 election: Requested vote from peers 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737), 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669)
I20260812 11:15:30.849018 6415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616320e73644258b82416d89f26ddef" candidate_uuid: "81e79624a9d040aa88f9ae538cc70621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18168ee7cdda412a90889a1be0714fc2"
I20260812 11:15:30.849038 6642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616320e73644258b82416d89f26ddef" candidate_uuid: "81e79624a9d040aa88f9ae538cc70621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0812419ccd80493bb22afa94d94683f6"
I20260812 11:15:30.849265 6642 raft_consensus.cc:3037] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:30.849265 6415 raft_consensus.cc:3037] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:30.851157 6642 raft_consensus.cc:2454] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e79624a9d040aa88f9ae538cc70621 in term 1.
I20260812 11:15:30.851157 6415 raft_consensus.cc:2454] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e79624a9d040aa88f9ae538cc70621 in term 1.
I20260812 11:15:30.851723 6778 leader_election.cc:302] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18168ee7cdda412a90889a1be0714fc2, 81e79624a9d040aa88f9ae538cc70621; no voters:
I20260812 11:15:30.852169 7131 raft_consensus.cc:2781] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:30.852567 7131 raft_consensus.cc:686] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 1 LEADER]: Becoming Leader. State: Replica: 81e79624a9d040aa88f9ae538cc70621, State: Running, Role: LEADER
I20260812 11:15:30.853013 7131 consensus_queue.cc:237] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } }
I20260812 11:15:30.853623 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.853287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:53749)
0812 11:15:30.853432 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.853524 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 11:15:30.856189 6209 catalog_manager.cc:5495] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "81e79624a9d040aa88f9ae538cc70621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81e79624a9d040aa88f9ae538cc70621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 46381 } health_report { overall_health: HEALTHY } } }
W20260812 11:15:30.857656 6970 tablet.cc:2369] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:30.879707 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.875148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.27.109.3:46381)
0812 11:15:30.875352 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:30.875359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:30.875378 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:30.875547 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:30.875550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:30.875635 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:30.875786 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:30.876102 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.876726 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.877730 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:30.877981 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:30.878109 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:30.878455 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:30.879209 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:30.879376 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:30.879417 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 11:15:30.879493 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:15:30.884341 6642 raft_consensus.cc:1261] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Refusing update from remote peer 81e79624a9d040aa88f9ae538cc70621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:30.884518 6415 raft_consensus.cc:1261] T c616320e73644258b82416d89f26ddef P 18168ee7cdda412a90889a1be0714fc2 [term 1 FOLLOWER]: Refusing update from remote peer 81e79624a9d040aa88f9ae538cc70621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:30.885216 7131 consensus_queue.cc:1035] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0812419ccd80493bb22afa94d94683f6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:30.885521 7135 consensus_queue.cc:1035] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18168ee7cdda412a90889a1be0714fc2" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:30.888721 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.888047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:42311)
0812 11:15:30.888495 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.888587 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260812 11:15:30.890043 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.889716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:46863)
0812 11:15:30.889848 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.889934 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 11:15:30.891037 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.890572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:45705)
0812 11:15:30.890776 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.890910 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 11:15:30.893394 7139 mvcc.cc:205] Tried to move back new op lower bound from 7317640523293302784 to 7317640523178127360. Current Snapshot: MvccSnapshot[applied={T|T < 7317640523293302784}]
I20260812 11:15:30.894942 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.894554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.2:43805)
0812 11:15:30.894715 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.894790 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 11:15:30.899415 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.899003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.1:54135)
0812 11:15:30.899216 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.899290 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 11:15:30.899581 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.899234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34707 (local address 127.27.109.3:46523)
0812 11:15:30.899414 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.899485 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 11:15:30.902593 6209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:30.902849 6209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:30.909219 6209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b65a1cd651f84a9d9e0004637f6e5e1b: Sending DeleteTablet for 3 replicas of tablet c616320e73644258b82416d89f26ddef
I20260812 11:15:30.910341 6606 tablet_service.cc:1514] Processing DeleteTablet for tablet c616320e73644258b82416d89f26ddef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:30 UTC) from {username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:38412
I20260812 11:15:30.910590 6856 tablet_service.cc:1514] Processing DeleteTablet for tablet c616320e73644258b82416d89f26ddef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:30 UTC) from {username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:39070
I20260812 11:15:30.910905 6385 tablet_service.cc:1514] Processing DeleteTablet for tablet c616320e73644258b82416d89f26ddef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:30 UTC) from {username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:60700
I20260812 11:15:30.911103 7152 tablet_replica.cc:330] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: stopping tablet replica
I20260812 11:15:30.911269 7152 raft_consensus.cc:2229] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:30.911754 7152 raft_consensus.cc:2258] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:30.912117 7151 tablet_replica.cc:330] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: stopping tablet replica
I20260812 11:15:30.912312 7151 raft_consensus.cc:2229] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:30.912520 7151 raft_consensus.cc:2258] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:30.912750 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 6140
I20260812 11:15:30.917608 7152 ts_tablet_manager.cc:1905] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:30.918044 7151 ts_tablet_manager.cc:1905] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:30.922103 7151 ts_tablet_manager.cc:1918] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:30.922118 7152 ts_tablet_manager.cc:1918] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:30.922245 7151 log.cc:1196] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/c616320e73644258b82416d89f26ddef
I20260812 11:15:30.922462 7152 log.cc:1196] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/c616320e73644258b82416d89f26ddef
I20260812 11:15:30.922665 7151 ts_tablet_manager.cc:1939] T c616320e73644258b82416d89f26ddef P 0812419ccd80493bb22afa94d94683f6: Deleting consensus metadata
I20260812 11:15:30.922996 7152 ts_tablet_manager.cc:1939] T c616320e73644258b82416d89f26ddef P 81e79624a9d040aa88f9ae538cc70621: Deleting consensus metadata
I20260812 11:15:30.924456 6185 catalog_manager.cc:4841] TS 0812419ccd80493bb22afa94d94683f6 (127.27.109.2:39669): tablet c616320e73644258b82416d89f26ddef (table test-table [id=dc98892f703f4315ae8795d507478ffc]) successfully deleted
I20260812 11:15:30.937708 6187 catalog_manager.cc:4841] TS 81e79624a9d040aa88f9ae538cc70621 (127.27.109.3:46381): tablet c616320e73644258b82416d89f26ddef (table test-table [id=dc98892f703f4315ae8795d507478ffc]) successfully deleted
I20260812 11:15:30.951663 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 6502
I20260812 11:15:30.951697 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:30.951206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37737 (local address 127.0.0.1:60716)
0812 11:15:30.951465 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:30.951550 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 11:15:30.955302 6190 catalog_manager.cc:4584] TS 18168ee7cdda412a90889a1be0714fc2 (127.27.109.1:37737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c616320e73644258b82416d89f26ddef: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37737: connect: Connection refused (error 111)
I20260812 11:15:30.969229 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 6738
I20260812 11:15:30.986999 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 5723
I20260812 11:15:30.999106 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 5875
2026-08-12T11:15:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9198 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:31 dist-test-slave-7z32 krb5kdc[7159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:31 dist-test-slave-7z32 krb5kdc[7159](info): set up 2 sockets
Aug 12 11:15:31 dist-test-slave-7z32 krb5kdc[7159](info): commencing operation
krb5kdc: starting...
W20260812 11:15:32.412201 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.368s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:32 dist-test-slave-7z32 krb5kdc[7159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
I20260812 11:15:32.584710 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38265
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:32.710544 7182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:32.710952 7182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:32.711066 7182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:32.715322 7182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:32.715399 7182 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:15:32.715416 7182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:32.715435 7182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:32.715453 7182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:32.715474 7182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:32.721252 7182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38265
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:32.722558 7182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:32.723739 7182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:32.734519 7190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:32.734864 7196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:32.734880 7192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:32.734949 7182 server_base.cc:1034] running on GCE node
I20260812 11:15:32.735747 7182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:32.737106 7182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:32.738394 7182 hybrid_clock.cc:648] HybridClock initialized: now 1786533332738372 us; error 46 us; skew 500 ppm
Aug 12 11:15:32 dist-test-slave-7z32 krb5kdc[7159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533332, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:32.742256 7182 init.cc:377] Logged in from keytab as oryx/127.27.109.62@KRBTEST.COM (short username oryx)
I20260812 11:15:32.743718 7182 webserver.cc:460] Webserver started at http://127.27.109.62:37329/ using document root <none> and password file <none>
I20260812 11:15:32.744484 7182 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:32.744604 7182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:32.744902 7182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:32.747469 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "179a35682bb14a7bbc68d875a66bae75"
format_stamp: "Formatted at 2026-08-12 11:15:32 on dist-test-slave-7z32"
server_key: "e67223003fb8f3ca3b8f5be0e9a99d65"
server_key_iv: "36633bffefb5b017e8c471eb0d469ca9"
server_key_version: "encryptionkey@0"
I20260812 11:15:32.748176 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "179a35682bb14a7bbc68d875a66bae75"
format_stamp: "Formatted at 2026-08-12 11:15:32 on dist-test-slave-7z32"
server_key: "e67223003fb8f3ca3b8f5be0e9a99d65"
server_key_iv: "36633bffefb5b017e8c471eb0d469ca9"
server_key_version: "encryptionkey@0"
I20260812 11:15:32.753192 7182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:32.756760 7206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:32.758702 7182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 11:15:32.759006 7182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "179a35682bb14a7bbc68d875a66bae75"
format_stamp: "Formatted at 2026-08-12 11:15:32 on dist-test-slave-7z32"
server_key: "e67223003fb8f3ca3b8f5be0e9a99d65"
server_key_iv: "36633bffefb5b017e8c471eb0d469ca9"
server_key_version: "encryptionkey@0"
I20260812 11:15:32.759178 7182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:32.790083 7182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:32.794533 7182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:32.794807 7182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:32.809297 7182 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38265
I20260812 11:15:32.809345 7306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38265 every 8 connection(s)
I20260812 11:15:32.810490 7182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:32.814528 7309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:32.817073 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 7182
I20260812 11:15:32.817255 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:32.817632 28084 external_mini_cluster.cc:1428] Setting key cc58092a1592d9e011a571cac383b74f
I20260812 11:15:32.823751 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Bootstrap starting.
I20260812 11:15:32.827080 7309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:32.828145 7309 log.cc:824] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:15:32 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:32.831779 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: No bootstrap required, opened a new log
I20260812 11:15:32.836238 7309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179a35682bb14a7bbc68d875a66bae75" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38265 } }
I20260812 11:15:32.836596 7309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:32.836750 7309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179a35682bb14a7bbc68d875a66bae75, State: Initialized, Role: FOLLOWER
I20260812 11:15:32.836624 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:32.819691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.27.109.62:38265)
0812 11:15:32.820478 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:32.820501 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:32.820555 (+ 54us) server_negotiation.cc:407] Connection header received
0812 11:15:32.821477 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:32.821499 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:32.821920 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:32.822322 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:32.823302 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:32.824454 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:32.825425 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:32.825976 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:32.826168 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:32.829360 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:32.829370 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:32.829395 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:32.829435 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:32.832695 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:32.833521 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:32.833530 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:32.833537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:32.833722 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:32.834058 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:32.834062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:32.834063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:32.834601 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:32.834913 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:32.835244 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 11:15:32.835635 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":203,"threads_started":1}
I20260812 11:15:32.837383 7309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179a35682bb14a7bbc68d875a66bae75" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38265 } }
I20260812 11:15:32.837579 7309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:32.837687 7309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:32.837829 7309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:32.839552 7309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179a35682bb14a7bbc68d875a66bae75" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38265 } }
I20260812 11:15:32.840090 7309 leader_election.cc:302] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179a35682bb14a7bbc68d875a66bae75; no voters:
I20260812 11:15:32.840590 7309 leader_election.cc:288] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:32.840843 7315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:32.841197 7315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [term 1 LEADER]: Becoming Leader. State: Replica: 179a35682bb14a7bbc68d875a66bae75, State: Running, Role: LEADER
I20260812 11:15:32.841619 7315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179a35682bb14a7bbc68d875a66bae75" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38265 } }
I20260812 11:15:32.842362 7309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:32.843765 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179a35682bb14a7bbc68d875a66bae75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179a35682bb14a7bbc68d875a66bae75" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38265 } } }
I20260812 11:15:32.844005 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:32.844578 7328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:32.844512 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179a35682bb14a7bbc68d875a66bae75. Latest consensus state: current_term: 1 leader_uuid: "179a35682bb14a7bbc68d875a66bae75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179a35682bb14a7bbc68d875a66bae75" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38265 } } }
I20260812 11:15:32.845968 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:32.849619 7328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:32.858973 7328 catalog_manager.cc:1347] Generated new cluster ID: 8d28df492f994f3cae6d82252a27b943
I20260812 11:15:32.859090 7328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:32.875074 7328 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:32.876219 7328 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:32.886636 7328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Generated new TSK 0
I20260812 11:15:32.887635 7328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
I20260812 11:15:32.940393 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38265
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:33.073809 7348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:33.074185 7348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:33.074241 7348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:33.079680 7348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:33.079780 7348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:33.079825 7348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:15:33.079867 7348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:33.086033 7348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:33.087376 7348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:33.088534 7348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:33.099866 7361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:33.100040 7362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:33.100260 7348 server_base.cc:1034] running on GCE node
W20260812 11:15:33.100337 7364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:33.100853 7348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:33.101675 7348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:33.105242 7348 hybrid_clock.cc:648] HybridClock initialized: now 1786533333105174 us; error 82 us; skew 500 ppm
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:33.109028 7348 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260812 11:15:33.110242 7348 webserver.cc:460] Webserver started at http://127.27.109.1:42999/ using document root <none> and password file <none>
I20260812 11:15:33.110988 7348 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:33.111073 7348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:33.111315 7348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:33.113893 7348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "6dbd3db10bcb46788c22747a6ff8a863"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "ab83a23d006f518fb58ea1cdd3088547"
server_key_iv: "da08fc1f8b3fea74fa58f3dd193c1c84"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.114646 7348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "6dbd3db10bcb46788c22747a6ff8a863"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "ab83a23d006f518fb58ea1cdd3088547"
server_key_iv: "da08fc1f8b3fea74fa58f3dd193c1c84"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.119520 7348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260812 11:15:33.122926 7376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:33.124420 7348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:33.124577 7348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "6dbd3db10bcb46788c22747a6ff8a863"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "ab83a23d006f518fb58ea1cdd3088547"
server_key_iv: "da08fc1f8b3fea74fa58f3dd193c1c84"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.124722 7348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:33.145807 7348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:33.149842 7348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:33.150072 7348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:33.152194 7348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:33.152262 7348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:33.152313 7348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:33.152379 7348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:33.167109 7348 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:44795
I20260812 11:15:33.167382 7558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:44795 every 8 connection(s)
I20260812 11:15:33.168308 7348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:33.170996 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 7348
I20260812 11:15:33.171231 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:33.171653 28084 external_mini_cluster.cc:1428] Setting key 81a988172a457ba59fa48be7f922af6d
I20260812 11:15:33.173484 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.153590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:32985 (local address 127.27.109.62:38265)
0812 11:15:33.153796 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.153801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.155385 (+ 1584us) server_negotiation.cc:407] Connection header received
0812 11:15:33.157185 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.157191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.157270 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.157419 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.160113 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.161079 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.163243 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.163667 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.163979 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.168108 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.168112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.168115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:33.168172 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.170295 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.171161 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.171168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.171170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.171319 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.171757 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.171761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.171764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.172085 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.172208 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.173077 (+ 869us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.173294 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:15:33.174304 7392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.154252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38265 (local address 127.27.109.1:32985)
0812 11:15:33.155083 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.155194 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.156825 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.157838 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.157848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.158651 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:33.159815 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.159850 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.161268 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.161274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.163062 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.163075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.163976 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.163981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.164276 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.165426 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:33.165448 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:33.167787 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:33.170515 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.170521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.170535 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.170983 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.171454 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.171456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.171458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.171619 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.172260 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:33.172268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:33.172690 (+ 422us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.172954 (+ 264us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.173354 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":184,"threads_started":1}
I20260812 11:15:33.176607 7559 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38265
I20260812 11:15:33.176857 7559 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:33.177897 7559 heartbeater.cc:511] Master 127.27.109.62:38265 requested a full tablet report, sending...
I20260812 11:15:33.180979 7229 ts_manager.cc:194] Registered new tserver with Master: 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795)
I20260812 11:15:33.183091 7229 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:32985
WARNING: no policy specified for oryx/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:33.192158 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.180842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:40965 (local address 127.27.109.62:38265)
0812 11:15:33.181007 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.181014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.181039 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:15:33.181171 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.181175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.181239 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.181357 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.182207 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.183344 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.185025 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.185531 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.185735 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.187754 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.187759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.187762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:33.187798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.189759 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.190431 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.190435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.190437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.190539 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.190937 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.190941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.190943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.191263 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.191400 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.191761 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.191932 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:15:33.192234 7568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.180256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38265 (local address 127.27.109.1:40965)
0812 11:15:33.180862 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.180888 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.181046 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.181419 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.181425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.181786 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:33.181987 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.181999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.183604 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.183609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.184708 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.184733 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.185757 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.185761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.185897 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.186930 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:33.186958 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:33.187482 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:33.189967 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.189972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.189975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.190268 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.190655 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.190659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.190662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.190777 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.191427 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:33.191431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:33.191555 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.191695 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.191981 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":220,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
I20260812 11:15:33.216269 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38265
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:33.340183 7571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:33.340550 7571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:33.340632 7571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:33.345333 7571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:33.345438 7571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:33.345517 7571 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:15:33.345602 7571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:33.351377 7571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:33.352579 7571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:33.353668 7571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:33.364537 7580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:33.364562 7577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:33.364533 7576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:33.365375 7571 server_base.cc:1034] running on GCE node
I20260812 11:15:33.365980 7571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:33.366935 7571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:33.368178 7571 hybrid_clock.cc:648] HybridClock initialized: now 1786533333368161 us; error 44 us; skew 500 ppm
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:33.372248 7571 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260812 11:15:33.373793 7571 webserver.cc:460] Webserver started at http://127.27.109.2:41339/ using document root <none> and password file <none>
I20260812 11:15:33.374650 7571 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:33.374774 7571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:33.375139 7571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:33.378407 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "e2d63485f6f247cbbe52c272e2795c64"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "cac1e009c89c492ae97a3ccc701a16a8"
server_key_iv: "0fe972361ceec420e80001ee4e70a57c"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.379217 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "e2d63485f6f247cbbe52c272e2795c64"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "cac1e009c89c492ae97a3ccc701a16a8"
server_key_iv: "0fe972361ceec420e80001ee4e70a57c"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.384830 7571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:15:33.388839 7593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:33.390691 7571 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 11:15:33.390980 7571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "e2d63485f6f247cbbe52c272e2795c64"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "cac1e009c89c492ae97a3ccc701a16a8"
server_key_iv: "0fe972361ceec420e80001ee4e70a57c"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.391148 7571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:33.411615 7571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:33.417037 7571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:33.417299 7571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:33.419739 7571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:33.419862 7571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:33.419942 7571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:33.419981 7571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:33.436794 7571 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:35899
I20260812 11:15:33.436815 7778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:35899 every 8 connection(s)
I20260812 11:15:33.438050 7571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:33.438736 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.420723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:38949 (local address 127.27.109.62:38265)
0812 11:15:33.420940 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.420946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.422054 (+ 1108us) server_negotiation.cc:407] Connection header received
0812 11:15:33.423312 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.423321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.423456 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.423632 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.425659 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.426740 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.428692 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.429235 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.429435 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.433620 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.433625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.433627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:33.433660 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.435812 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.436818 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.436828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.436830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.436961 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.437405 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.437409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.437411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.437652 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.437773 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.438397 (+ 624us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.438553 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 11:15:33.439844 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.421166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38265 (local address 127.27.109.2:38949)
0812 11:15:33.421819 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.421877 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.422944 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.423858 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.423868 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.424492 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:33.425377 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.425404 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.427101 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.427105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.428433 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.428447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.429481 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.429489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.429777 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.430675 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:33.430701 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:33.433347 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:33.436057 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.436063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.436082 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.436673 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.437104 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.437109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.437111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.437295 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.437771 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:33.437780 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:33.438102 (+ 322us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.438353 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.438752 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":248,"threads_started":1}
I20260812 11:15:33.441526 7779 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38265
I20260812 11:15:33.441751 7779 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:33.442667 7779 heartbeater.cc:511] Master 127.27.109.62:38265 requested a full tablet report, sending...
I20260812 11:15:33.444211 7225 ts_manager.cc:194] Registered new tserver with Master: e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899)
I20260812 11:15:33.445119 7225 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:38949
I20260812 11:15:33.447050 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 7571
I20260812 11:15:33.447180 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:33.447530 28084 external_mini_cluster.cc:1428] Setting key e0ebca23e2b66300c35016e65a303c82
I20260812 11:15:33.455031 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.444222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:60025 (local address 127.27.109.62:38265)
0812 11:15:33.444404 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.444410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.444797 (+ 387us) server_negotiation.cc:407] Connection header received
0812 11:15:33.444894 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.444897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.444971 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.445097 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.445691 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.446622 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.447947 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.448336 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.448456 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.449947 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.449952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.449956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:33.449992 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.452462 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.453205 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.453208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.453210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.453331 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.453734 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.453737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.453739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.454054 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.454201 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.454602 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.454752 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:15:33.455070 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.444179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38265 (local address 127.27.109.2:60025)
0812 11:15:33.444645 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.444663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.444789 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.445147 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.445150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.445394 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:33.445571 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.445579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.446798 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.446803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.447786 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.447799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.448493 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.448497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.448594 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.449388 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:33.449400 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:33.449760 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:33.452655 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.452658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.452659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.453043 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.453458 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.453464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.453467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.453590 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.454215 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:33.454218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:33.454384 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.454516 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.454713 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":177,"threads_started":1}
WARNING: no policy specified for oryx/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
I20260812 11:15:33.486945 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38265
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:33.645745 7793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:33.646147 7793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:33.646237 7793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:33.650988 7793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:33.651108 7793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:33.651156 7793 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:15:33.651226 7793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:33.657814 7793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:33.659623 7793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:33.661033 7793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:33.674063 7799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:33.674093 7798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:33.674077 7805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:33.674690 7793 server_base.cc:1034] running on GCE node
I20260812 11:15:33.675566 7793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:33.676790 7793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:33.678067 7793 hybrid_clock.cc:648] HybridClock initialized: now 1786533333678025 us; error 60 us; skew 500 ppm
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:33.682438 7793 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260812 11:15:33.684240 7793 webserver.cc:460] Webserver started at http://127.27.109.3:40833/ using document root <none> and password file <none>
I20260812 11:15:33.685169 7793 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:33.685266 7793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:33.685586 7793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:33.688676 7793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "13d9f07740f846efbbd9850b5369a1e3"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "5d60e0c3e600e7ad45010e6fa74b45c2"
server_key_iv: "173e9ec949d26de81787aedbab318b14"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.689481 7793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "13d9f07740f846efbbd9850b5369a1e3"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "5d60e0c3e600e7ad45010e6fa74b45c2"
server_key_iv: "173e9ec949d26de81787aedbab318b14"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.695869 7793 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 11:15:33.700304 7820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:33.702595 7793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:15:33.702941 7793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "13d9f07740f846efbbd9850b5369a1e3"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-7z32"
server_key: "5d60e0c3e600e7ad45010e6fa74b45c2"
server_key_iv: "173e9ec949d26de81787aedbab318b14"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.703161 7793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:33.718032 7793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:33.723071 7793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:33.723358 7793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:33.725987 7793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:33.726125 7793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:33.726207 7793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:33.726243 7793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:33.742653 7793 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38219
I20260812 11:15:33.742760 7999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38219 every 8 connection(s)
I20260812 11:15:33.744315 7793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:33.745307 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.726738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37281 (local address 127.27.109.62:38265)
0812 11:15:33.726989 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.726992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.728794 (+ 1802us) server_negotiation.cc:407] Connection header received
0812 11:15:33.730206 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.730210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.730277 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.730393 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.732248 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.733367 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.735134 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.735394 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.735522 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.740599 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.740604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.740607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:33.740643 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.742638 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.743348 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.743352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.743354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.743447 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.743862 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.743865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.743867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.744104 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.744251 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.744994 (+ 743us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.745132 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:15:33.746356 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.727640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38265 (local address 127.27.109.3:37281)
0812 11:15:33.728539 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.728622 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.729936 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.730761 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.730772 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.731279 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:33.731998 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.732019 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.733602 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.733607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.734969 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.734987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.735574 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.735579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.735787 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.736671 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:33.736710 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:33.740345 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:33.742785 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.742795 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.742841 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.743209 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.743547 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.743550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.743553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.743743 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.744251 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:33.744261 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:33.744696 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.744936 (+ 240us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.745327 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":229,"threads_started":1}
I20260812 11:15:33.748306 8000 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38265
I20260812 11:15:33.748508 8000 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:33.749244 8000 heartbeater.cc:511] Master 127.27.109.62:38265 requested a full tablet report, sending...
I20260812 11:15:33.749604 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 7793
I20260812 11:15:33.749831 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:33.750270 28084 external_mini_cluster.cc:1428] Setting key 774acae9cc2acd876f2b24458d616fe8
I20260812 11:15:33.751912 7229 ts_manager.cc:194] Registered new tserver with Master: 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:33.752882 7229 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:37281
I20260812 11:15:33.754199 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:33.761317 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.751244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:54723 (local address 127.27.109.62:38265)
0812 11:15:33.751372 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.751376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.751966 (+ 590us) server_negotiation.cc:407] Connection header received
0812 11:15:33.752108 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.752111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.752164 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.752266 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.752853 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.753622 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.754730 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.755159 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.755265 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.756569 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.756573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.756576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:33.756602 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.758752 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.759523 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.759529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.759532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.759693 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.760083 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.760087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.760089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.760389 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.760488 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.760993 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.761153 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 11:15:33.761418 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.751277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38265 (local address 127.27.109.3:54723)
0812 11:15:33.751856 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.751876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.752009 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.752289 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.752292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.752557 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:33.752699 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.752706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.753791 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.753794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.754586 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.754597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.755282 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.755286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.755386 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.755990 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:33.756002 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:33.756408 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:33.759022 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.759026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.759028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.759373 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.759843 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:33.759846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:33.759848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:33.759955 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:33.760514 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:33.760520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:33.760672 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.760810 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.761057 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":164,"threads_started":1}
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:15:33.790637 7313 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:33.781897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.27.109.62:38265)
0812 11:15:33.782073 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.782077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.782097 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:15:33.782153 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.782156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.782207 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.782309 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.783398 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.784049 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.785444 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.785704 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.785821 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.790454 (+ 4633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37314: BlockingRecv error: recv got EOF from 127.0.0.1:37314 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 11:15:33 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:33.804881 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.793016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.27.109.62:38265)
0812 11:15:33.793184 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.793188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.793334 (+ 146us) server_negotiation.cc:407] Connection header received
0812 11:15:33.793455 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.793458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.793517 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.793620 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.794452 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.795194 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.796368 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.796816 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.796940 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.800248 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.800252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.800255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:33.800304 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.802158 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.802809 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.802812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.802814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.803027 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:33.803649 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:33.803652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:33.803654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:33.803948 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.804118 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.804459 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.804703 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:15:33.810274 7225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:33.814618 7225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:33.835391 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.829094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.27.109.1:44795)
0812 11:15:33.829627 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.830000 (+ 373us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.830031 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:15:33.830131 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.830140 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.830401 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.830572 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:33.830890 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.832078 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.833930 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.834506 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.834659 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.834873 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.835111 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.835220 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":81,"threads_started":1}
I20260812 11:15:33.835391 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.828346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44795 (local address 127.0.0.1:33596)
0812 11:15:33.829777 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.829795 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.829903 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.830578 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.830581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.830605 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:33.830738 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.830746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.832238 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.832242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.833745 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.833760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.834668 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.834672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.834858 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.834914 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.835064 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.835221 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1363,"mutex_wait_us":108,"thread_start_us":118,"threads_started":1}
I20260812 11:15:33.836254 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.828972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38219 (local address 127.0.0.1:36508)
0812 11:15:33.830092 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.830108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.830229 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.830943 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.830947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.830967 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:33.831091 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.831098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.832456 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.832460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.833766 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.833773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.835597 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.835602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.835767 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.835798 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.835887 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.836022 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":79,"thread_start_us":100,"threads_started":1}
I20260812 11:15:33.836730 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.828305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.27.109.2:35899)
0812 11:15:33.829201 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.829219 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.829251 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:15:33.829510 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.829517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.829837 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.830042 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:33.830455 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.831699 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.833685 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.835051 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.835253 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.835504 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.836354 (+ 850us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.836541 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":214,"threads_started":1}
I20260812 11:15:33.837083 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.828065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:35899 (local address 127.0.0.1:49710)
0812 11:15:33.828959 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.829043 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.829250 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.830080 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.830088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.830113 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:33.830306 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.830313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.831846 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.831850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.833525 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.833538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.835597 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.835602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:33.836042 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.836065 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:15:33.836158 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 11:15:33.836238 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":46,"thread_start_us":129,"threads_started":1}
I20260812 11:15:33.837550 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.829465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.27.109.3:38219)
0812 11:15:33.829904 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.829918 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.830347 (+ 429us) server_negotiation.cc:407] Connection header received
0812 11:15:33.830426 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.830431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.830596 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.830734 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:33.831242 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.832328 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.833951 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:33.835213 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:33.835334 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:33.835503 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.836603 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.836763 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":137,"threads_started":1}
I20260812 11:15:33.839306 7451 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcb0771426c43d28bde8828ca80b815 (DEFAULT_TABLE table=test-table [id=49bc8425adae4978830f5a7d44042697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:33.840557 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcb0771426c43d28bde8828ca80b815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:33.841090 7895 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcb0771426c43d28bde8828ca80b815 (DEFAULT_TABLE table=test-table [id=49bc8425adae4978830f5a7d44042697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:33.841089 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcb0771426c43d28bde8828ca80b815 (DEFAULT_TABLE table=test-table [id=49bc8425adae4978830f5a7d44042697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:33.842660 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcb0771426c43d28bde8828ca80b815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:33.842705 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcb0771426c43d28bde8828ca80b815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:33.848623 8045 tablet_bootstrap.cc:492] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Bootstrap starting.
I20260812 11:15:33.851384 8045 tablet_bootstrap.cc:654] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:33.852502 8045 log.cc:824] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:33.854200 8052 tablet_bootstrap.cc:492] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Bootstrap starting.
I20260812 11:15:33.854882 8047 tablet_bootstrap.cc:492] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Bootstrap starting.
I20260812 11:15:33.857049 8047 tablet_bootstrap.cc:654] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:33.857049 8052 tablet_bootstrap.cc:654] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:33.857659 8045 tablet_bootstrap.cc:492] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: No bootstrap required, opened a new log
I20260812 11:15:33.857970 8045 ts_tablet_manager.cc:1397] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260812 11:15:33.858047 8052 log.cc:824] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:33.858047 8047 log.cc:824] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:33.860692 8047 tablet_bootstrap.cc:492] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: No bootstrap required, opened a new log
I20260812 11:15:33.860942 8047 ts_tablet_manager.cc:1397] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:33.861012 8052 tablet_bootstrap.cc:492] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: No bootstrap required, opened a new log
I20260812 11:15:33.861284 8052 ts_tablet_manager.cc:1397] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:15:33.863917 8045 raft_consensus.cc:348] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:33.864394 8045 raft_consensus.cc:374] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:33.864470 8045 raft_consensus.cc:729] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbd3db10bcb46788c22747a6ff8a863, State: Initialized, Role: FOLLOWER
I20260812 11:15:33.864542 8047 raft_consensus.cc:348] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:33.864837 8047 raft_consensus.cc:374] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:33.864919 8047 raft_consensus.cc:729] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d9f07740f846efbbd9850b5369a1e3, State: Initialized, Role: FOLLOWER
I20260812 11:15:33.865128 8045 consensus_queue.cc:260] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:33.865474 8047 consensus_queue.cc:260] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:33.865579 8052 raft_consensus.cc:348] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:33.865844 8052 raft_consensus.cc:374] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:33.865901 8052 raft_consensus.cc:729] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Initialized, Role: FOLLOWER
I20260812 11:15:33.866287 7559 heartbeater.cc:503] Master 127.27.109.62:38265 was elected leader, sending a full tablet report...
I20260812 11:15:33.866359 8052 consensus_queue.cc:260] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:33.866482 8045 ts_tablet_manager.cc:1428] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Time spent starting tablet: real 0.008s user 0.005s sys 0.005s
I20260812 11:15:33.866642 8047 ts_tablet_manager.cc:1428] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:33.867233 8000 heartbeater.cc:503] Master 127.27.109.62:38265 was elected leader, sending a full tablet report...
I20260812 11:15:33.867791 7779 heartbeater.cc:503] Master 127.27.109.62:38265 was elected leader, sending a full tablet report...
I20260812 11:15:33.868654 8052 ts_tablet_manager.cc:1428] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260812 11:15:33.920107 7560 tablet.cc:2369] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:15:33.940197 7780 tablet.cc:2369] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:33.995311 8058 raft_consensus.cc:482] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:33.995609 8058 raft_consensus.cc:504] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
W20260812 11:15:33.996301 8002 tablet.cc:2369] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:33.997314 8058 leader_election.cc:288] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.003554 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.998524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:40399 (local address 127.27.109.3:38219)
0812 11:15:33.998708 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.998715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.998912 (+ 197us) server_negotiation.cc:407] Connection header received
0812 11:15:33.999120 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.999124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.999170 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.999288 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:33.999562 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.000286 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.002130 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.002783 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.002984 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.003030 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.003280 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.003372 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:15:34.003568 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.998316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38219 (local address 127.27.109.2:40399)
0812 11:15:33.998779 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.998796 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.998991 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.999316 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.999318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.999336 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:33.999436 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.999440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.000396 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.000399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.001971 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.001979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.002988 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.002992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.003094 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.003150 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.003230 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.003323 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":80,"thread_start_us":96,"threads_started":1}
I20260812 11:15:34.004166 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.997692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44795 (local address 127.27.109.2:46489)
0812 11:15:33.998127 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:33.998151 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:33.998366 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:33.998801 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:33.998807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:33.998887 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:33.999109 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:33.999125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.000225 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.000228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.001964 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.001979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.003161 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.003165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.003344 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.003381 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.003486 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.003606 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":87,"thread_start_us":150,"threads_started":1}
I20260812 11:15:34.004206 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.997941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:46489 (local address 127.27.109.1:44795)
0812 11:15:33.998158 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.998167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.998317 (+ 150us) server_negotiation.cc:407] Connection header received
0812 11:15:33.998578 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.998584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.998645 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.998739 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:33.999253 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.000083 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.002130 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.002783 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.002966 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.003030 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.003563 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.003802 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 11:15:34.004844 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcb0771426c43d28bde8828ca80b815" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3" is_pre_election: true
I20260812 11:15:34.004973 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcb0771426c43d28bde8828ca80b815" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863" is_pre_election: true
I20260812 11:15:34.005383 7487 raft_consensus.cc:2454] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 0.
I20260812 11:15:34.005383 7930 raft_consensus.cc:2454] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 0.
I20260812 11:15:34.006187 7596 leader_election.cc:302] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9f07740f846efbbd9850b5369a1e3, e2d63485f6f247cbbe52c272e2795c64; no voters:
I20260812 11:15:34.006606 8058 raft_consensus.cc:2781] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:34.006769 8058 raft_consensus.cc:482] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:34.006832 8058 raft_consensus.cc:3037] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.008368 8058 raft_consensus.cc:504] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.008944 8058 leader_election.cc:288] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 election: Requested vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.009717 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcb0771426c43d28bde8828ca80b815" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863"
I20260812 11:15:34.009720 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcb0771426c43d28bde8828ca80b815" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3"
I20260812 11:15:34.009939 7487 raft_consensus.cc:3037] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.009939 7930 raft_consensus.cc:3037] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.012413 7487 raft_consensus.cc:2454] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 1.
I20260812 11:15:34.012817 7930 raft_consensus.cc:2454] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 1.
I20260812 11:15:34.013211 7596 leader_election.cc:302] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbd3db10bcb46788c22747a6ff8a863, e2d63485f6f247cbbe52c272e2795c64; no voters:
I20260812 11:15:34.013485 8058 raft_consensus.cc:2781] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:34.013798 8058 raft_consensus.cc:686] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 1 LEADER]: Becoming Leader. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Running, Role: LEADER
I20260812 11:15:34.014307 8058 consensus_queue.cc:237] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.019364 7225 catalog_manager.cc:5495] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "e2d63485f6f247cbbe52c272e2795c64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:34.043082 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.038309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.27.109.2:35899)
0812 11:15:34.038472 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.038476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.038494 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:34.038611 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.038614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.038690 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.038883 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:34.039273 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.039883 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.040896 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.041249 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.041356 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.041675 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.042449 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.042554 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.042729 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.042815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:15:34.051846 7930 raft_consensus.cc:1261] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Refusing update from remote peer e2d63485f6f247cbbe52c272e2795c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.051998 7487 raft_consensus.cc:1261] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Refusing update from remote peer e2d63485f6f247cbbe52c272e2795c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.052983 8058 consensus_queue.cc:1035] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.053287 8062 consensus_queue.cc:1035] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.063457 8067 mvcc.cc:205] Tried to move back new op lower bound from 7317640536264183808 to 7317640536126709760. Current Snapshot: MvccSnapshot[applied={T|T < 7317640536264183808}]
I20260812 11:15:34.064955 8068 mvcc.cc:205] Tried to move back new op lower bound from 7317640536264183808 to 7317640536126709760. Current Snapshot: MvccSnapshot[applied={T|T < 7317640536264183808}]
I20260812 11:15:34.075374 7225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:34.075574 7225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:34.078898 7225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Sending DeleteTablet for 3 replicas of tablet 8dcb0771426c43d28bde8828ca80b815
I20260812 11:15:34.080358 7895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcb0771426c43d28bde8828ca80b815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36508
I20260812 11:15:34.080320 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcb0771426c43d28bde8828ca80b815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49710
I20260812 11:15:34.080644 7451 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcb0771426c43d28bde8828ca80b815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:33596
I20260812 11:15:34.081126 8085 tablet_replica.cc:330] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: stopping tablet replica
I20260812 11:15:34.081341 8086 tablet_replica.cc:330] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: stopping tablet replica
I20260812 11:15:34.081650 8085 raft_consensus.cc:2229] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:34.081794 8086 raft_consensus.cc:2229] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:34.082170 8085 raft_consensus.cc:2258] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.082231 8086 raft_consensus.cc:2258] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.082160 7225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:15:34.083137 8082 tablet_replica.cc:330] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: stopping tablet replica
W20260812 11:15:34.082814 7225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:34.083650 8082 raft_consensus.cc:2229] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:34.084259 8082 raft_consensus.cc:2258] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.084910 8086 ts_tablet_manager.cc:1905] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.085814 8085 ts_tablet_manager.cc:1905] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.089426 8086 ts_tablet_manager.cc:1918] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:34.090844 7895 tablet_service.cc:1467] Processing CreateTablet for tablet e96406e7e37444649d1816c99df8485a (DEFAULT_TABLE table=test-table [id=37599f1654ef4a9aab0184c4bfa224f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.091259 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96406e7e37444649d1816c99df8485a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.091187 7676 tablet_service.cc:1467] Processing CreateTablet for tablet e96406e7e37444649d1816c99df8485a (DEFAULT_TABLE table=test-table [id=37599f1654ef4a9aab0184c4bfa224f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.091639 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96406e7e37444649d1816c99df8485a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.093001 8082 ts_tablet_manager.cc:1905] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.094492 8047 tablet_bootstrap.cc:492] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: Bootstrap starting.
I20260812 11:15:34.094669 8052 tablet_bootstrap.cc:492] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: Bootstrap starting.
I20260812 11:15:34.095757 8047 tablet_bootstrap.cc:654] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.095968 8052 tablet_bootstrap.cc:654] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.095934 8082 ts_tablet_manager.cc:1918] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:34.096743 8082 log.cc:1196] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/8dcb0771426c43d28bde8828ca80b815
I20260812 11:15:34.090407 7451 tablet_service.cc:1467] Processing CreateTablet for tablet e96406e7e37444649d1816c99df8485a (DEFAULT_TABLE table=test-table [id=37599f1654ef4a9aab0184c4bfa224f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.097689 8047 tablet_bootstrap.cc:492] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: No bootstrap required, opened a new log
I20260812 11:15:34.097795 8047 ts_tablet_manager.cc:1397] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:34.097923 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96406e7e37444649d1816c99df8485a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.098188 8082 ts_tablet_manager.cc:1939] T 8dcb0771426c43d28bde8828ca80b815 P e2d63485f6f247cbbe52c272e2795c64: Deleting consensus metadata
I20260812 11:15:34.098452 8047 raft_consensus.cc:348] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.098603 8047 raft_consensus.cc:374] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.098644 8047 raft_consensus.cc:729] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d9f07740f846efbbd9850b5369a1e3, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.098896 8047 consensus_queue.cc:260] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.099782 8047 ts_tablet_manager.cc:1428] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:34.100003 7215 catalog_manager.cc:4841] TS e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899): tablet 8dcb0771426c43d28bde8828ca80b815 (table test-table [id=49bc8425adae4978830f5a7d44042697]) successfully deleted
I20260812 11:15:34.100868 8052 tablet_bootstrap.cc:492] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: No bootstrap required, opened a new log
I20260812 11:15:34.100977 8052 ts_tablet_manager.cc:1397] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 11:15:34.101536 8052 raft_consensus.cc:348] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.101676 8052 raft_consensus.cc:374] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.101718 8052 raft_consensus.cc:729] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.101814 8086 log.cc:1196] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/8dcb0771426c43d28bde8828ca80b815
I20260812 11:15:34.101872 8052 consensus_queue.cc:260] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.102186 8052 ts_tablet_manager.cc:1428] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:15:34.102286 8086 ts_tablet_manager.cc:1939] T 8dcb0771426c43d28bde8828ca80b815 P 6dbd3db10bcb46788c22747a6ff8a863: Deleting consensus metadata
I20260812 11:15:34.102500 8045 tablet_bootstrap.cc:492] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: Bootstrap starting.
I20260812 11:15:34.103960 7212 catalog_manager.cc:4841] TS 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795): tablet 8dcb0771426c43d28bde8828ca80b815 (table test-table [id=49bc8425adae4978830f5a7d44042697]) successfully deleted
I20260812 11:15:34.090863 8085 ts_tablet_manager.cc:1918] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:34.104475 8085 log.cc:1196] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/8dcb0771426c43d28bde8828ca80b815
I20260812 11:15:34.104900 8085 ts_tablet_manager.cc:1939] T 8dcb0771426c43d28bde8828ca80b815 P 13d9f07740f846efbbd9850b5369a1e3: Deleting consensus metadata
I20260812 11:15:34.104951 8045 tablet_bootstrap.cc:654] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.106390 7212 catalog_manager.cc:4841] TS 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219): tablet 8dcb0771426c43d28bde8828ca80b815 (table test-table [id=49bc8425adae4978830f5a7d44042697]) successfully deleted
I20260812 11:15:34.106881 8045 tablet_bootstrap.cc:492] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: No bootstrap required, opened a new log
I20260812 11:15:34.107026 8045 ts_tablet_manager.cc:1397] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:15:34.108234 8045 raft_consensus.cc:348] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.108378 8045 raft_consensus.cc:374] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.108403 8045 raft_consensus.cc:729] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbd3db10bcb46788c22747a6ff8a863, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.108624 8045 consensus_queue.cc:260] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.108997 8045 ts_tablet_manager.cc:1428] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 11:15:34.150888 8062 raft_consensus.cc:482] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:34.151111 8062 raft_consensus.cc:504] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.151718 8062 leader_election.cc:288] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.152561 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96406e7e37444649d1816c99df8485a" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863" is_pre_election: true
I20260812 11:15:34.152566 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96406e7e37444649d1816c99df8485a" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3" is_pre_election: true
I20260812 11:15:34.152765 7487 raft_consensus.cc:2454] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 0.
I20260812 11:15:34.152913 7930 raft_consensus.cc:2454] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 0.
I20260812 11:15:34.153234 7596 leader_election.cc:302] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbd3db10bcb46788c22747a6ff8a863, e2d63485f6f247cbbe52c272e2795c64; no voters:
I20260812 11:15:34.153524 8062 raft_consensus.cc:2781] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:34.153626 8062 raft_consensus.cc:482] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:34.153683 8062 raft_consensus.cc:3037] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.155200 8062 raft_consensus.cc:504] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.155694 8062 leader_election.cc:288] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 election: Requested vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.156378 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96406e7e37444649d1816c99df8485a" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3"
I20260812 11:15:34.156402 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96406e7e37444649d1816c99df8485a" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863"
I20260812 11:15:34.156558 7930 raft_consensus.cc:3037] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.156610 7487 raft_consensus.cc:3037] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.157835 7930 raft_consensus.cc:2454] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 1.
I20260812 11:15:34.158149 7487 raft_consensus.cc:2454] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 1.
I20260812 11:15:34.158308 7596 leader_election.cc:302] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9f07740f846efbbd9850b5369a1e3, e2d63485f6f247cbbe52c272e2795c64; no voters:
I20260812 11:15:34.158686 8062 raft_consensus.cc:2781] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:34.158818 8062 raft_consensus.cc:686] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 1 LEADER]: Becoming Leader. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Running, Role: LEADER
I20260812 11:15:34.159016 8062 consensus_queue.cc:237] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.161417 7229 catalog_manager.cc:5495] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "e2d63485f6f247cbbe52c272e2795c64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } health_report { overall_health: UNKNOWN } } }
Aug 12 11:15:34 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533332, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:34.201884 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.189677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.27.109.62:38265)
0812 11:15:34.189868 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.189874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.190190 (+ 316us) server_negotiation.cc:407] Connection header received
0812 11:15:34.190376 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.190379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.190421 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.190507 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:34.191550 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.192201 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.193063 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.193288 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.193402 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.196720 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:34.196724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:34.196726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:34.196780 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:34.198936 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.200186 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.200192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.200194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.200295 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.200606 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.200611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.200613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.200857 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:34.200978 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.201342 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.201526 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":64}
I20260812 11:15:34.201956 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.189585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38265 (local address 127.0.0.1:37344)
0812 11:15:34.190027 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.190043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.190243 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.190533 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.190536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.191178 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:34.191375 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.191383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.192324 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.192328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.192934 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.192943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.193419 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.193423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.193558 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.194402 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:34.194417 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:34.196499 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:34.199469 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:34.199492 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:34.199512 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:34.200043 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:34.200402 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:34.200407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:34.200409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:34.200501 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:34.200978 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:34.200990 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:34.201120 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.201293 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.201656 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":180,"thread_start_us":197,"threads_started":1}
I20260812 11:15:34.209812 7229 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:37344:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:15:34.211026 7229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:34.218117 7451 tablet_service.cc:1467] Processing CreateTablet for tablet 1d382c32ff764bb88004eed4d0231241 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35c1d9e850284c3e9ee1ba0c31789270]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:15:34.218763 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d382c32ff764bb88004eed4d0231241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.218571 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 1d382c32ff764bb88004eed4d0231241 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35c1d9e850284c3e9ee1ba0c31789270]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:15:34.219153 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d382c32ff764bb88004eed4d0231241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.222373 8045 tablet_bootstrap.cc:492] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: Bootstrap starting.
I20260812 11:15:34.223290 7895 tablet_service.cc:1467] Processing CreateTablet for tablet 1d382c32ff764bb88004eed4d0231241 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35c1d9e850284c3e9ee1ba0c31789270]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:15:34.223739 8045 tablet_bootstrap.cc:654] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.223815 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d382c32ff764bb88004eed4d0231241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.225595 8045 tablet_bootstrap.cc:492] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: No bootstrap required, opened a new log
I20260812 11:15:34.225734 8045 ts_tablet_manager.cc:1397] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:34.226261 8045 raft_consensus.cc:348] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.226419 8045 raft_consensus.cc:374] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.226464 8045 raft_consensus.cc:729] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbd3db10bcb46788c22747a6ff8a863, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.226660 8045 consensus_queue.cc:260] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.227005 8055 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.227334 8055 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:15:34.227548 8047 tablet_bootstrap.cc:492] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: Bootstrap starting.
I20260812 11:15:34.227674 8045 ts_tablet_manager.cc:1428] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:34.228847 8047 tablet_bootstrap.cc:654] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.229713 8052 tablet_bootstrap.cc:492] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: Bootstrap starting.
I20260812 11:15:34.231074 8052 tablet_bootstrap.cc:654] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.232733 8052 tablet_bootstrap.cc:492] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: No bootstrap required, opened a new log
I20260812 11:15:34.232782 8047 tablet_bootstrap.cc:492] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: No bootstrap required, opened a new log
I20260812 11:15:34.232884 8052 ts_tablet_manager.cc:1397] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:34.232892 8047 ts_tablet_manager.cc:1397] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 11:15:34.233454 8047 raft_consensus.cc:348] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.233522 8052 raft_consensus.cc:348] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.233628 8047 raft_consensus.cc:374] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.233675 8052 raft_consensus.cc:374] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.233685 8047 raft_consensus.cc:729] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d9f07740f846efbbd9850b5369a1e3, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.233726 8052 raft_consensus.cc:729] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.233924 8052 consensus_queue.cc:260] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.234081 8047 consensus_queue.cc:260] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.234252 8062 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.234411 8062 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:15:34.234414 8057 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.234601 8057 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:15:34.234714 8052 ts_tablet_manager.cc:1428] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:34.236011 8047 ts_tablet_manager.cc:1428] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:34.269603 8057 raft_consensus.cc:482] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:34.269917 8057 raft_consensus.cc:504] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.271718 8057 leader_election.cc:288] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899)
I20260812 11:15:34.277758 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.272115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:35899 (local address 127.27.109.3:58959)
0812 11:15:34.272815 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.272835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.273009 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.273452 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.273455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.273474 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:34.273599 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.273606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.274795 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.274801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.276084 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.276093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.276987 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.276991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.277174 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.277201 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.277388 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.277505 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":157,"threads_started":1}
I20260812 11:15:34.277716 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.272477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:58959 (local address 127.27.109.2:35899)
0812 11:15:34.272651 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.272668 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.272928 (+ 260us) server_negotiation.cc:407] Connection header received
0812 11:15:34.273173 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.273177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.273242 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.273299 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:34.273802 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.274612 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.276243 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.276826 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.276991 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.277075 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.277415 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.277513 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:15:34.277807 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.272602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:56723 (local address 127.27.109.1:44795)
0812 11:15:34.272788 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.272793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.272811 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:34.272903 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.272910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.272984 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.273087 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:34.273583 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.274267 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.276100 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.276832 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.276987 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.277049 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.277491 (+ 442us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.277596 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 11:15:34.278327 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.272000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44795 (local address 127.27.109.3:56723)
0812 11:15:34.272518 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.272533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.272666 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.273247 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.273250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.273271 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:34.273437 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.273445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.274406 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.274408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.275892 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.275905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.277147 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.277152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.277318 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.277354 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.278014 (+ 660us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.278136 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":85,"threads_started":1}
I20260812 11:15:34.278759 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d382c32ff764bb88004eed4d0231241" candidate_uuid: "13d9f07740f846efbbd9850b5369a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d63485f6f247cbbe52c272e2795c64" is_pre_election: true
I20260812 11:15:34.278992 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d382c32ff764bb88004eed4d0231241" candidate_uuid: "13d9f07740f846efbbd9850b5369a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863" is_pre_election: true
I20260812 11:15:34.279212 7487 raft_consensus.cc:2454] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d9f07740f846efbbd9850b5369a1e3 in term 0.
I20260812 11:15:34.279191 7704 raft_consensus.cc:2454] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d9f07740f846efbbd9850b5369a1e3 in term 0.
I20260812 11:15:34.279772 7821 leader_election.cc:302] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9f07740f846efbbd9850b5369a1e3, 6dbd3db10bcb46788c22747a6ff8a863; no voters:
I20260812 11:15:34.280196 8057 raft_consensus.cc:2781] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:34.280296 8057 raft_consensus.cc:482] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:34.280344 8057 raft_consensus.cc:3037] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.281780 8057 raft_consensus.cc:504] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.282284 8057 leader_election.cc:288] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [CANDIDATE]: Term 1 election: Requested vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899)
I20260812 11:15:34.282881 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d382c32ff764bb88004eed4d0231241" candidate_uuid: "13d9f07740f846efbbd9850b5369a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863"
I20260812 11:15:34.282913 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d382c32ff764bb88004eed4d0231241" candidate_uuid: "13d9f07740f846efbbd9850b5369a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d63485f6f247cbbe52c272e2795c64"
I20260812 11:15:34.283090 7487 raft_consensus.cc:3037] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.283109 7704 raft_consensus.cc:3037] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.284356 7487 raft_consensus.cc:2454] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d9f07740f846efbbd9850b5369a1e3 in term 1.
I20260812 11:15:34.284585 7704 raft_consensus.cc:2454] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d9f07740f846efbbd9850b5369a1e3 in term 1.
I20260812 11:15:34.284808 7821 leader_election.cc:302] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9f07740f846efbbd9850b5369a1e3, 6dbd3db10bcb46788c22747a6ff8a863; no voters:
I20260812 11:15:34.285084 8057 raft_consensus.cc:2781] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:34.285343 8057 raft_consensus.cc:686] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 LEADER]: Becoming Leader. State: Replica: 13d9f07740f846efbbd9850b5369a1e3, State: Running, Role: LEADER
I20260812 11:15:34.285837 8057 consensus_queue.cc:237] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.287619 8114 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: TxnStatusTablet state changed. Reason: New leader 13d9f07740f846efbbd9850b5369a1e3. Latest consensus state: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.287806 8114 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:15:34.288218 8117 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:15:34.288432 8117 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:15:34.289186 7231 catalog_manager.cc:5495] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } health_report { overall_health: HEALTHY } } }
I20260812 11:15:34.316349 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.310818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38219 (local address 127.0.0.1:36514)
0812 11:15:34.311093 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.311115 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.311271 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.311803 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.311807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.311816 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:15:34.311946 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.311951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.313130 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.313133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.313861 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.313873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.314552 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.314555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.314628 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.314704 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.315933 (+ 1229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.315955 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.316041 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.316142 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 11:15:34.316375 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.311043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.27.109.3:38219)
0812 11:15:34.311248 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.311255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.311279 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:15:34.311530 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.311534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.311632 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.311767 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:34.312068 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.312938 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.314019 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.314390 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.314514 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.314950 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.315779 (+ 829us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.315928 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.316090 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.316177 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:15:34.320456 7487 raft_consensus.cc:1261] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Refusing update from remote peer 13d9f07740f846efbbd9850b5369a1e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.320559 7704 raft_consensus.cc:1261] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Refusing update from remote peer 13d9f07740f846efbbd9850b5369a1e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.321153 8114 consensus_queue.cc:1035] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.321357 8057 consensus_queue.cc:1035] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.323184 8055 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: TxnStatusTablet state changed. Reason: New leader 13d9f07740f846efbbd9850b5369a1e3. Latest consensus state: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.323313 8055 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:15:34.323966 8062 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: TxnStatusTablet state changed. Reason: New leader 13d9f07740f846efbbd9850b5369a1e3. Latest consensus state: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.324087 8062 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:15:34.326442 8057 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.326644 8057 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:15:34.327147 8114 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.327384 8114 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P 13d9f07740f846efbbd9850b5369a1e3: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:15:34.328622 8062 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.328800 8062 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P e2d63485f6f247cbbe52c272e2795c64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:15:34.330576 8055 tablet_replica.cc:439] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13d9f07740f846efbbd9850b5369a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } } }
I20260812 11:15:34.330758 8055 tablet_replica.cc:442] T 1d382c32ff764bb88004eed4d0231241 P 6dbd3db10bcb46788c22747a6ff8a863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:15:34.346933 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.342134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:37831 (local address 127.27.109.3:38219)
0812 11:15:34.342335 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.342342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.342608 (+ 266us) server_negotiation.cc:407] Connection header received
0812 11:15:34.342782 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.342786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.342882 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.343009 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:34.343455 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.344339 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.345495 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.345865 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.345999 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.346329 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.346449 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.346533 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.346641 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.346714 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:15:34.346942 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.341935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38219 (local address 127.27.109.2:37831)
0812 11:15:34.342478 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.342493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.342651 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.343066 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.343069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.343077 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:15:34.343305 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.343313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.344487 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.344490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.345365 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.345376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.346017 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.346020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.346103 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.346163 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.346554 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.346560 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.346624 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.346710 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":157,"threads_started":1}
I20260812 11:15:34.356585 7930 raft_consensus.cc:1261] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Refusing update from remote peer e2d63485f6f247cbbe52c272e2795c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.357045 7487 raft_consensus.cc:1261] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Refusing update from remote peer e2d63485f6f247cbbe52c272e2795c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.357553 8062 consensus_queue.cc:1035] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.357764 8058 consensus_queue.cc:1035] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.390070 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.384429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:36277 (local address 127.27.109.2:35899)
0812 11:15:34.384627 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.384634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.385162 (+ 528us) server_negotiation.cc:407] Connection header received
0812 11:15:34.385274 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.385279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.385432 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.385521 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:34.385953 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.387018 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.388077 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.388478 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.388625 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.389224 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.389402 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.389514 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.389697 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.389856 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:15:34.390457 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.384186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:35899 (local address 127.27.109.3:36277)
0812 11:15:34.384946 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.384967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.385156 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.385547 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.385552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.385563 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:15:34.385758 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.385765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.387190 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.387194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.387922 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.387933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.388686 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.388689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.388847 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.388946 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.389518 (+ 572us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.389527 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.389619 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.389734 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":226,"threads_started":1}
I20260812 11:15:34.438678 7231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:34.438910 7231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:34.441176 7231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Sending DeleteTablet for 3 replicas of tablet e96406e7e37444649d1816c99df8485a
I20260812 11:15:34.441982 7895 tablet_service.cc:1514] Processing DeleteTablet for tablet e96406e7e37444649d1816c99df8485a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36508
I20260812 11:15:34.442116 7451 tablet_service.cc:1514] Processing DeleteTablet for tablet e96406e7e37444649d1816c99df8485a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:33596
I20260812 11:15:34.442236 8085 tablet_replica.cc:330] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: stopping tablet replica
I20260812 11:15:34.442395 8085 raft_consensus.cc:2229] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:34.442430 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet e96406e7e37444649d1816c99df8485a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49710
I20260812 11:15:34.442528 8086 tablet_replica.cc:330] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: stopping tablet replica
I20260812 11:15:34.442551 8085 raft_consensus.cc:2258] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.442730 8086 raft_consensus.cc:2229] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:34.442950 8086 raft_consensus.cc:2258] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.443168 8082 tablet_replica.cc:330] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: stopping tablet replica
I20260812 11:15:34.443357 8082 raft_consensus.cc:2229] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:34.443610 8082 raft_consensus.cc:2258] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.444099 8085 ts_tablet_manager.cc:1905] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.444774 8082 ts_tablet_manager.cc:1905] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.447638 8085 ts_tablet_manager.cc:1918] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:15:34.447722 8085 log.cc:1196] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/e96406e7e37444649d1816c99df8485a
I20260812 11:15:34.448318 8085 ts_tablet_manager.cc:1939] T e96406e7e37444649d1816c99df8485a P 13d9f07740f846efbbd9850b5369a1e3: Deleting consensus metadata
I20260812 11:15:34.448873 8082 ts_tablet_manager.cc:1918] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:15:34.449007 8082 log.cc:1196] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/e96406e7e37444649d1816c99df8485a
I20260812 11:15:34.449319 8082 ts_tablet_manager.cc:1939] T e96406e7e37444649d1816c99df8485a P e2d63485f6f247cbbe52c272e2795c64: Deleting consensus metadata
I20260812 11:15:34.449498 7212 catalog_manager.cc:4841] TS 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219): tablet e96406e7e37444649d1816c99df8485a (table test-table [id=37599f1654ef4a9aab0184c4bfa224f0]) successfully deleted
I20260812 11:15:34.450402 7215 catalog_manager.cc:4841] TS e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899): tablet e96406e7e37444649d1816c99df8485a (table test-table [id=37599f1654ef4a9aab0184c4bfa224f0]) successfully deleted
I20260812 11:15:34.456837 8086 ts_tablet_manager.cc:1905] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.461513 8086 ts_tablet_manager.cc:1918] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:15:34.461687 8086 log.cc:1196] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/e96406e7e37444649d1816c99df8485a
I20260812 11:15:34.462023 8086 ts_tablet_manager.cc:1939] T e96406e7e37444649d1816c99df8485a P 6dbd3db10bcb46788c22747a6ff8a863: Deleting consensus metadata
I20260812 11:15:34.463402 7212 catalog_manager.cc:4841] TS 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795): tablet e96406e7e37444649d1816c99df8485a (table test-table [id=37599f1654ef4a9aab0184c4bfa224f0]) successfully deleted
Aug 12 11:15:34 dist-test-slave-7z32 krb5kdc[7159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:15:34 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:34 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:15:34.490511 7313 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:34.481024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.27.109.62:38265)
0812 11:15:34.481200 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.481204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.481600 (+ 396us) server_negotiation.cc:407] Connection header received
0812 11:15:34.481739 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.481742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.481799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.481994 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:34.483088 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.483706 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.485095 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.485458 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.485574 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.490351 (+ 4777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37358: BlockingRecv error: recv got EOF from 127.0.0.1:37358 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 11:15:34 dist-test-slave-7z32 krb5kdc[7159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260812 11:15:34.507556 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.493132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.27.109.62:38265)
0812 11:15:34.493331 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.493337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.493502 (+ 165us) server_negotiation.cc:407] Connection header received
0812 11:15:34.493628 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.493631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.493706 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.493834 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:34.494739 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.495667 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.497209 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.497634 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.497802 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.501373 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:34.501377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:34.501379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:34.501455 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:34.504858 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.505904 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.505909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.505911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.506076 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.506431 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.506435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.506437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.506689 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:34.506892 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.507205 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.507356 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:15:34.513135 7231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:34.514201 7231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:34.521598 7895 tablet_service.cc:1467] Processing CreateTablet for tablet 9c61a5d62f484b1a9aaefed91da08aba (DEFAULT_TABLE table=test-table [id=b17079fbac864bbc8dc58b943b421fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.521584 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 9c61a5d62f484b1a9aaefed91da08aba (DEFAULT_TABLE table=test-table [id=b17079fbac864bbc8dc58b943b421fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.521554 7451 tablet_service.cc:1467] Processing CreateTablet for tablet 9c61a5d62f484b1a9aaefed91da08aba (DEFAULT_TABLE table=test-table [id=b17079fbac864bbc8dc58b943b421fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.521970 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c61a5d62f484b1a9aaefed91da08aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.521970 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c61a5d62f484b1a9aaefed91da08aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.522068 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c61a5d62f484b1a9aaefed91da08aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.525034 8047 tablet_bootstrap.cc:492] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: Bootstrap starting.
I20260812 11:15:34.525419 8045 tablet_bootstrap.cc:492] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: Bootstrap starting.
I20260812 11:15:34.526165 8052 tablet_bootstrap.cc:492] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: Bootstrap starting.
I20260812 11:15:34.526187 8047 tablet_bootstrap.cc:654] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.526809 8045 tablet_bootstrap.cc:654] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.527441 8052 tablet_bootstrap.cc:654] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.527997 8047 tablet_bootstrap.cc:492] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: No bootstrap required, opened a new log
I20260812 11:15:34.528105 8047 ts_tablet_manager.cc:1397] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:34.528622 8045 tablet_bootstrap.cc:492] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: No bootstrap required, opened a new log
I20260812 11:15:34.528743 8045 ts_tablet_manager.cc:1397] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:34.528718 8047 raft_consensus.cc:348] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.528867 8047 raft_consensus.cc:374] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.528918 8047 raft_consensus.cc:729] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d9f07740f846efbbd9850b5369a1e3, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.528944 8052 tablet_bootstrap.cc:492] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: No bootstrap required, opened a new log
I20260812 11:15:34.529040 8052 ts_tablet_manager.cc:1397] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:34.529261 8047 consensus_queue.cc:260] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.529409 8045 raft_consensus.cc:348] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.529584 8045 raft_consensus.cc:374] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.529505 8052 raft_consensus.cc:348] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.529624 8045 raft_consensus.cc:729] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbd3db10bcb46788c22747a6ff8a863, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.529639 8052 raft_consensus.cc:374] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.529695 8052 raft_consensus.cc:729] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.529806 8045 consensus_queue.cc:260] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.529840 8052 consensus_queue.cc:260] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.530376 8045 ts_tablet_manager.cc:1428] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:34.530390 8052 ts_tablet_manager.cc:1428] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:15:34.531458 8047 ts_tablet_manager.cc:1428] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 11:15:34.850258 8055 raft_consensus.cc:482] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:34.850492 8055 raft_consensus.cc:504] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.852231 8055 leader_election.cc:288] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.858103 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.852818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:36131 (local address 127.27.109.2:35899)
0812 11:15:34.852992 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.852998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.853381 (+ 383us) server_negotiation.cc:407] Connection header received
0812 11:15:34.853492 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.853495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.853555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.853647 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:34.854027 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.855025 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.856785 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.857302 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.857499 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.857588 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.857795 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.857912 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:15:34.858146 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.852714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:35899 (local address 127.27.109.1:36131)
0812 11:15:34.853275 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.853292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.853388 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.853710 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.853713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.853732 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:34.853863 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.853869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.855218 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.855222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.856594 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.856607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.857486 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.857491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.857609 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.857653 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.857773 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.857894 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":95,"thread_start_us":105,"threads_started":1}
I20260812 11:15:34.858213 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.852705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38219 (local address 127.27.109.1:39415)
0812 11:15:34.853172 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:34.853191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:34.853370 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:34.854059 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:34.854062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:34.854082 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:34.854739 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.854750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.855815 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.855820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.856835 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:34.856850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.857629 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.857634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:34.857759 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.857787 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:15:34.857897 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 11:15:34.858009 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":112,"threads_started":1}
I20260812 11:15:34.858197 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.852897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:39415 (local address 127.27.109.3:38219)
0812 11:15:34.853215 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.853219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.853302 (+ 83us) server_negotiation.cc:407] Connection header received
0812 11:15:34.853815 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.853817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.853861 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.853956 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:34.854961 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.855634 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.857003 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.857475 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.857629 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.857698 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.857920 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.858008 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260812 11:15:34.859061 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61a5d62f484b1a9aaefed91da08aba" candidate_uuid: "6dbd3db10bcb46788c22747a6ff8a863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3" is_pre_election: true
I20260812 11:15:34.859257 7930 raft_consensus.cc:2454] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbd3db10bcb46788c22747a6ff8a863 in term 0.
I20260812 11:15:34.859246 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61a5d62f484b1a9aaefed91da08aba" candidate_uuid: "6dbd3db10bcb46788c22747a6ff8a863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d63485f6f247cbbe52c272e2795c64" is_pre_election: true
I20260812 11:15:34.859489 7704 raft_consensus.cc:2454] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbd3db10bcb46788c22747a6ff8a863 in term 0.
I20260812 11:15:34.859822 7377 leader_election.cc:302] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9f07740f846efbbd9850b5369a1e3, 6dbd3db10bcb46788c22747a6ff8a863; no voters:
I20260812 11:15:34.860369 8055 raft_consensus.cc:2781] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:34.860461 8055 raft_consensus.cc:482] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:34.860495 8055 raft_consensus.cc:3037] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.861968 8055 raft_consensus.cc:504] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.862466 8055 leader_election.cc:288] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [CANDIDATE]: Term 1 election: Requested vote from peers e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.863251 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61a5d62f484b1a9aaefed91da08aba" candidate_uuid: "6dbd3db10bcb46788c22747a6ff8a863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d63485f6f247cbbe52c272e2795c64"
I20260812 11:15:34.863251 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61a5d62f484b1a9aaefed91da08aba" candidate_uuid: "6dbd3db10bcb46788c22747a6ff8a863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3"
I20260812 11:15:34.863469 7704 raft_consensus.cc:3037] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.863474 7930 raft_consensus.cc:3037] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.865127 7704 raft_consensus.cc:2454] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbd3db10bcb46788c22747a6ff8a863 in term 1.
I20260812 11:15:34.865116 7930 raft_consensus.cc:2454] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbd3db10bcb46788c22747a6ff8a863 in term 1.
I20260812 11:15:34.865581 7377 leader_election.cc:302] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13d9f07740f846efbbd9850b5369a1e3, 6dbd3db10bcb46788c22747a6ff8a863, e2d63485f6f247cbbe52c272e2795c64; no voters:
I20260812 11:15:34.865876 8055 raft_consensus.cc:2781] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:34.866166 8055 raft_consensus.cc:686] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbd3db10bcb46788c22747a6ff8a863, State: Running, Role: LEADER
I20260812 11:15:34.866547 8055 consensus_queue.cc:237] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.869870 7229 catalog_manager.cc:5495] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "6dbd3db10bcb46788c22747a6ff8a863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:34.912441 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.906985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.27.109.1:44795)
0812 11:15:34.907628 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.907636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.907654 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:34.907757 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.907759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.907882 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.908096 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:34.908457 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.909159 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.910222 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.910541 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.910676 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.911126 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:34.911732 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:34.912016 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.912074 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.912148 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":255,"threads_started":1}
I20260812 11:15:34.916069 7704 raft_consensus.cc:1261] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Refusing update from remote peer 6dbd3db10bcb46788c22747a6ff8a863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.916168 7930 raft_consensus.cc:1261] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Refusing update from remote peer 6dbd3db10bcb46788c22747a6ff8a863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:34.916940 8175 consensus_queue.cc:1035] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.917266 8055 consensus_queue.cc:1035] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:34.935541 7229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:34.935778 7229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:34.938632 7229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Sending DeleteTablet for 3 replicas of tablet 9c61a5d62f484b1a9aaefed91da08aba
I20260812 11:15:34.939601 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c61a5d62f484b1a9aaefed91da08aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49710
I20260812 11:15:34.939692 7895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c61a5d62f484b1a9aaefed91da08aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36508
I20260812 11:15:34.939903 8085 tablet_replica.cc:330] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: stopping tablet replica
I20260812 11:15:34.939915 8082 tablet_replica.cc:330] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: stopping tablet replica
I20260812 11:15:34.940064 8085 raft_consensus.cc:2229] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:34.940064 8082 raft_consensus.cc:2229] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:34.940240 8085 raft_consensus.cc:2258] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.940236 8082 raft_consensus.cc:2258] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.940538 7451 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c61a5d62f484b1a9aaefed91da08aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:34 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:33596
I20260812 11:15:34.940769 8086 tablet_replica.cc:330] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: stopping tablet replica
I20260812 11:15:34.940892 8086 raft_consensus.cc:2229] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:34.941354 8086 raft_consensus.cc:2258] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:34.941799 8082 ts_tablet_manager.cc:1905] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.942001 7229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:15:34.942442 8086 ts_tablet_manager.cc:1905] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 11:15:34.942625 7229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:34.946069 8086 ts_tablet_manager.cc:1918] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:34.946208 8086 log.cc:1196] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/9c61a5d62f484b1a9aaefed91da08aba
I20260812 11:15:34.946493 8082 ts_tablet_manager.cc:1918] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:34.946627 8086 ts_tablet_manager.cc:1939] T 9c61a5d62f484b1a9aaefed91da08aba P 6dbd3db10bcb46788c22747a6ff8a863: Deleting consensus metadata
I20260812 11:15:34.946775 8082 log.cc:1196] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/9c61a5d62f484b1a9aaefed91da08aba
I20260812 11:15:34.947213 8082 ts_tablet_manager.cc:1939] T 9c61a5d62f484b1a9aaefed91da08aba P e2d63485f6f247cbbe52c272e2795c64: Deleting consensus metadata
I20260812 11:15:34.947933 7212 catalog_manager.cc:4841] TS 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795): tablet 9c61a5d62f484b1a9aaefed91da08aba (table test-table [id=b17079fbac864bbc8dc58b943b421fec]) successfully deleted
I20260812 11:15:34.948341 8085 ts_tablet_manager.cc:1905] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:34.949595 7676 tablet_service.cc:1467] Processing CreateTablet for tablet b976ac7bef9c438fad1e6c6d0a45e609 (DEFAULT_TABLE table=test-table [id=f789374e3198486fa930b6269923e4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.949862 7895 tablet_service.cc:1467] Processing CreateTablet for tablet b976ac7bef9c438fad1e6c6d0a45e609 (DEFAULT_TABLE table=test-table [id=f789374e3198486fa930b6269923e4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.950139 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b976ac7bef9c438fad1e6c6d0a45e609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.951790 7451 tablet_service.cc:1467] Processing CreateTablet for tablet b976ac7bef9c438fad1e6c6d0a45e609 (DEFAULT_TABLE table=test-table [id=f789374e3198486fa930b6269923e4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:34.952171 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b976ac7bef9c438fad1e6c6d0a45e609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.952746 8047 tablet_bootstrap.cc:492] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: Bootstrap starting.
I20260812 11:15:34.955291 8045 tablet_bootstrap.cc:492] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863: Bootstrap starting.
I20260812 11:15:34.955371 8047 tablet_bootstrap.cc:654] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.956175 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b976ac7bef9c438fad1e6c6d0a45e609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.956805 8045 tablet_bootstrap.cc:654] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.957396 8085 ts_tablet_manager.cc:1918] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:34.957517 8085 log.cc:1196] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/9c61a5d62f484b1a9aaefed91da08aba
I20260812 11:15:34.957844 8085 ts_tablet_manager.cc:1939] T 9c61a5d62f484b1a9aaefed91da08aba P 13d9f07740f846efbbd9850b5369a1e3: Deleting consensus metadata
I20260812 11:15:34.959021 8045 tablet_bootstrap.cc:492] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863: No bootstrap required, opened a new log
I20260812 11:15:34.959156 8045 ts_tablet_manager.cc:1397] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:34.959653 8052 tablet_bootstrap.cc:492] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: Bootstrap starting.
I20260812 11:15:34.959759 8047 tablet_bootstrap.cc:492] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: No bootstrap required, opened a new log
I20260812 11:15:34.959861 8047 ts_tablet_manager.cc:1397] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 11:15:34.959834 8045 raft_consensus.cc:348] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.959862 7212 catalog_manager.cc:4841] TS 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219): tablet 9c61a5d62f484b1a9aaefed91da08aba (table test-table [id=b17079fbac864bbc8dc58b943b421fec]) successfully deleted
I20260812 11:15:34.960016 8045 raft_consensus.cc:374] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.960054 8045 raft_consensus.cc:729] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbd3db10bcb46788c22747a6ff8a863, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.960219 8045 consensus_queue.cc:260] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.960335 8047 raft_consensus.cc:348] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.960500 8047 raft_consensus.cc:374] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.960553 8047 raft_consensus.cc:729] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d9f07740f846efbbd9850b5369a1e3, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.960717 8052 tablet_bootstrap.cc:654] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:34.960728 8045 ts_tablet_manager.cc:1428] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:34.960740 8047 consensus_queue.cc:260] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.961104 8047 ts_tablet_manager.cc:1428] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:15:34.963282 8052 tablet_bootstrap.cc:492] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: No bootstrap required, opened a new log
I20260812 11:15:34.963434 8052 ts_tablet_manager.cc:1397] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260812 11:15:34.963943 8052 raft_consensus.cc:348] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.964113 8052 raft_consensus.cc:374] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.964156 8052 raft_consensus.cc:729] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.964309 8052 consensus_queue.cc:260] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.964658 8052 ts_tablet_manager.cc:1428] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:15:34.965647 7215 catalog_manager.cc:4841] TS e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899): tablet 9c61a5d62f484b1a9aaefed91da08aba (table test-table [id=b17079fbac864bbc8dc58b943b421fec]) successfully deleted
I20260812 11:15:34.981863 8136 raft_consensus.cc:482] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:34.982226 8136 raft_consensus.cc:504] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.982877 8136 leader_election.cc:288] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.983798 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b976ac7bef9c438fad1e6c6d0a45e609" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3" is_pre_election: true
I20260812 11:15:34.983798 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b976ac7bef9c438fad1e6c6d0a45e609" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863" is_pre_election: true
I20260812 11:15:34.984062 7930 raft_consensus.cc:2454] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 0.
I20260812 11:15:34.984062 7487 raft_consensus.cc:2454] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 0.
I20260812 11:15:34.984529 7596 leader_election.cc:302] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbd3db10bcb46788c22747a6ff8a863, e2d63485f6f247cbbe52c272e2795c64; no voters:
I20260812 11:15:34.984853 8136 raft_consensus.cc:2781] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:34.984935 8136 raft_consensus.cc:482] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:34.984973 8136 raft_consensus.cc:3037] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.986430 8136 raft_consensus.cc:504] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.986948 8136 leader_election.cc:288] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 election: Requested vote from peers 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795), 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219)
I20260812 11:15:34.987545 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b976ac7bef9c438fad1e6c6d0a45e609" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbd3db10bcb46788c22747a6ff8a863"
I20260812 11:15:34.987773 7487 raft_consensus.cc:3037] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.987725 7930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b976ac7bef9c438fad1e6c6d0a45e609" candidate_uuid: "e2d63485f6f247cbbe52c272e2795c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9f07740f846efbbd9850b5369a1e3"
I20260812 11:15:34.987897 7930 raft_consensus.cc:3037] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.989748 7487 raft_consensus.cc:2454] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 1.
I20260812 11:15:34.989748 7930 raft_consensus.cc:2454] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d63485f6f247cbbe52c272e2795c64 in term 1.
I20260812 11:15:34.990298 7596 leader_election.cc:302] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9f07740f846efbbd9850b5369a1e3, e2d63485f6f247cbbe52c272e2795c64; no voters:
I20260812 11:15:34.990646 8136 raft_consensus.cc:2781] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:34.991010 8136 raft_consensus.cc:686] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 1 LEADER]: Becoming Leader. State: Replica: e2d63485f6f247cbbe52c272e2795c64, State: Running, Role: LEADER
I20260812 11:15:34.991237 8136 consensus_queue.cc:237] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } }
I20260812 11:15:34.993453 7229 catalog_manager.cc:5495] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "e2d63485f6f247cbbe52c272e2795c64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2d63485f6f247cbbe52c272e2795c64" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:35.026593 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:35.022641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.27.109.2:35899)
0812 11:15:35.022810 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:35.022814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:35.022887 (+ 73us) server_negotiation.cc:407] Connection header received
0812 11:15:35.022963 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:35.022965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:35.023023 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:35.023132 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:35.023433 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:35.024015 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:35.024979 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:35.025289 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:35.025447 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:35.025941 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:35.026051 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:35.026152 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:35.026300 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 11:15:35.026381 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:15:35.037179 7487 raft_consensus.cc:1261] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Refusing update from remote peer e2d63485f6f247cbbe52c272e2795c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:35.037181 7930 raft_consensus.cc:1261] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Refusing update from remote peer e2d63485f6f247cbbe52c272e2795c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:35.037833 8136 consensus_queue.cc:1035] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbd3db10bcb46788c22747a6ff8a863" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:35.038103 8195 consensus_queue.cc:1035] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d9f07740f846efbbd9850b5369a1e3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:35.087021 7231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:35.087285 7231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:35.090241 7231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179a35682bb14a7bbc68d875a66bae75: Sending DeleteTablet for 3 replicas of tablet b976ac7bef9c438fad1e6c6d0a45e609
I20260812 11:15:35.091265 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet b976ac7bef9c438fad1e6c6d0a45e609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:35 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49710
I20260812 11:15:35.091563 7451 tablet_service.cc:1514] Processing DeleteTablet for tablet b976ac7bef9c438fad1e6c6d0a45e609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:35 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:33596
I20260812 11:15:35.091804 8082 tablet_replica.cc:330] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: stopping tablet replica
I20260812 11:15:35.091864 7895 tablet_service.cc:1514] Processing DeleteTablet for tablet b976ac7bef9c438fad1e6c6d0a45e609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:35 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36508
I20260812 11:15:35.091975 8082 raft_consensus.cc:2229] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:35.092180 8085 tablet_replica.cc:330] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: stopping tablet replica
I20260812 11:15:35.092300 8082 raft_consensus.cc:2258] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:35.092325 8085 raft_consensus.cc:2229] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:35.092480 8085 raft_consensus.cc:2258] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:35.092516 8086 tablet_replica.cc:330] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863: stopping tablet replica
I20260812 11:15:35.092679 8086 raft_consensus.cc:2229] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:35.092698 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 7348
I20260812 11:15:35.092823 8086 raft_consensus.cc:2258] T b976ac7bef9c438fad1e6c6d0a45e609 P 6dbd3db10bcb46788c22747a6ff8a863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:35.106665 8085 ts_tablet_manager.cc:1905] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:35.107803 8082 ts_tablet_manager.cc:1905] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:35.111877 8082 ts_tablet_manager.cc:1918] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:15:35.112032 8082 log.cc:1196] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/b976ac7bef9c438fad1e6c6d0a45e609
I20260812 11:15:35.112342 8082 ts_tablet_manager.cc:1939] T b976ac7bef9c438fad1e6c6d0a45e609 P e2d63485f6f247cbbe52c272e2795c64: Deleting consensus metadata
I20260812 11:15:35.113026 8085 ts_tablet_manager.cc:1918] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:15:35.113173 8085 log.cc:1196] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/b976ac7bef9c438fad1e6c6d0a45e609
I20260812 11:15:35.113415 7215 catalog_manager.cc:4841] TS e2d63485f6f247cbbe52c272e2795c64 (127.27.109.2:35899): tablet b976ac7bef9c438fad1e6c6d0a45e609 (table test-table [id=f789374e3198486fa930b6269923e4d7]) successfully deleted
I20260812 11:15:35.113484 8085 ts_tablet_manager.cc:1939] T b976ac7bef9c438fad1e6c6d0a45e609 P 13d9f07740f846efbbd9850b5369a1e3: Deleting consensus metadata
I20260812 11:15:35.115015 7212 catalog_manager.cc:4841] TS 13d9f07740f846efbbd9850b5369a1e3 (127.27.109.3:38219): tablet b976ac7bef9c438fad1e6c6d0a45e609 (table test-table [id=f789374e3198486fa930b6269923e4d7]) successfully deleted
W20260812 11:15:35.122570 7212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:44795 (error 108)
I20260812 11:15:35.122745 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 7571
I20260812 11:15:35.141656 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:35.139480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44795 (local address 127.0.0.1:33608)
0812 11:15:35.141466 (+ 1986us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:35.141549 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:44795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1897,"thread_start_us":1706,"threads_started":1}
W20260812 11:15:35.142014 7212 catalog_manager.cc:4584] TS 6dbd3db10bcb46788c22747a6ff8a863 (127.27.109.1:44795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b976ac7bef9c438fad1e6c6d0a45e609: Network error: Client connection negotiation failed: client connection to 127.27.109.1:44795: connect: Connection refused (error 111)
I20260812 11:15:35.153810 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 7793
I20260812 11:15:35.176419 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 7182
2026-08-12T11:15:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4174 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:39 dist-test-slave-7z32 krb5kdc[8215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:39 dist-test-slave-7z32 krb5kdc[8215](info): set up 2 sockets
Aug 12 11:15:39 dist-test-slave-7z32 krb5kdc[8215](info): commencing operation
krb5kdc: starting...
W20260812 11:15:41.122136 28084 mini_kdc.cc:121] Time spent starting KDC: real 5.903s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:41 dist-test-slave-7z32 krb5kdc[8215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:41.312142 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35845
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:35845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:41.469691 8236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:41.470145 8236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:41.470198 8236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:41.475009 8236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:41.475134 8236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:41.475158 8236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:41.475179 8236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:41.475198 8236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:41.482975 8236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:35845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:41.485682 8236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:41.487443 8236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:41.500618 8236 server_base.cc:1034] running on GCE node
W20260812 11:15:41.500656 8245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:41.500653 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:41.501026 8247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:41.501973 8236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:41.503629 8236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:41.505002 8236 hybrid_clock.cc:648] HybridClock initialized: now 1786533341504976 us; error 100 us; skew 500 ppm
W20260812 11:15:42.935007 8236 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-7z32': real 1.429s user 0.002s sys 0.000s
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T11:15:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (7749 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:43 dist-test-slave-7z32 krb5kdc[8254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:43 dist-test-slave-7z32 krb5kdc[8254](info): set up 2 sockets
Aug 12 11:15:43 dist-test-slave-7z32 krb5kdc[8254](info): commencing operation
krb5kdc: starting...
W20260812 11:15:44.345578 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.377s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:15:44 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:15:44.540272 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36075
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:44.700402 8277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:44.700829 8277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:44.700922 8277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:44.706614 8277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:44.706790 8277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:44.706833 8277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:44.706892 8277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:44.706930 8277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:44.715210 8277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:44.717201 8277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:44.718752 8277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:44.731514 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:44.731832 8286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:44.732123 8277 server_base.cc:1034] running on GCE node
W20260812 11:15:44.731534 8292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:44.733034 8277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:44.734992 8277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:44.736321 8277 hybrid_clock.cc:648] HybridClock initialized: now 1786533344736319 us; error 122 us; skew 500 ppm
Aug 12 11:15:44 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:44.740516 8277 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:44.742008 8277 webserver.cc:460] Webserver started at http://127.27.109.62:39037/ using document root <none> and password file <none>
I20260812 11:15:44.743176 8277 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:44.743287 8277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:44.743551 8277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:44.747200 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "0f4053ba429d4c4a881aeb0af8941bd1"
format_stamp: "Formatted at 2026-08-12 11:15:44 on dist-test-slave-7z32"
server_key: "540e6c7d1530e4348cf3808e3937b066"
server_key_iv: "d908a093255aec0b7bd860999cb1b068"
server_key_version: "encryptionkey@0"
I20260812 11:15:44.748116 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "0f4053ba429d4c4a881aeb0af8941bd1"
format_stamp: "Formatted at 2026-08-12 11:15:44 on dist-test-slave-7z32"
server_key: "540e6c7d1530e4348cf3808e3937b066"
server_key_iv: "d908a093255aec0b7bd860999cb1b068"
server_key_version: "encryptionkey@0"
I20260812 11:15:44.754653 8277 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:15:44.759029 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:44.761483 8277 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:15:44.761804 8277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0f4053ba429d4c4a881aeb0af8941bd1"
format_stamp: "Formatted at 2026-08-12 11:15:44 on dist-test-slave-7z32"
server_key: "540e6c7d1530e4348cf3808e3937b066"
server_key_iv: "d908a093255aec0b7bd860999cb1b068"
server_key_version: "encryptionkey@0"
I20260812 11:15:44.762034 8277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:44.794812 8277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:44.800099 8277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:44.800459 8277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:44.812122 8277 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:36075
I20260812 11:15:44.812242 8394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:36075 every 8 connection(s)
I20260812 11:15:44.813884 8277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:44.817967 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:44.824195 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 8277
I20260812 11:15:44.824369 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:15:44.824716 28084 external_mini_cluster.cc:1428] Setting key 7e2446573f1ace1ea6d9aaa4131d9a4c
I20260812 11:15:44.826424 8395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Bootstrap starting.
I20260812 11:15:44.830188 8395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:44.831707 8395 log.cc:824] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:44.836258 8395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: No bootstrap required, opened a new log
Aug 12 11:15:44 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:44.843542 8395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:44.843962 8395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:44.844040 8395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4053ba429d4c4a881aeb0af8941bd1, State: Initialized, Role: FOLLOWER
I20260812 11:15:44.845160 8395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:44.845484 8395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:44.845623 8395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:44.845759 8395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:44.847260 8395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:44.847755 8395 leader_election.cc:302] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f4053ba429d4c4a881aeb0af8941bd1; no voters:
I20260812 11:15:44.848261 8395 leader_election.cc:288] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:44.848464 8405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:44.848810 8405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 0f4053ba429d4c4a881aeb0af8941bd1, State: Running, Role: LEADER
I20260812 11:15:44.849335 8405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:44.849947 8395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:44.851243 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:44.827185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.27.109.62:36075)
0812 11:15:44.827908 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:44.827932 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:44.827987 (+ 55us) server_negotiation.cc:407] Connection header received
0812 11:15:44.829027 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:44.829058 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:44.829589 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:44.830026 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:44.831708 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:44.833450 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:44.835496 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:44.836633 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:44.836929 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:44.841258 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:44.841269 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:44.841300 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:44.841358 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:44.844763 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:44.846063 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:44.846073 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:44.846084 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:44.846321 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:44.846990 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:44.846994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:44.846995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:44.847897 (+ 902us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:44.848217 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:44.849168 (+ 951us) server_negotiation.cc:299] Negotiation successful
0812 11:15:44.849818 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":234,"threads_started":1}
I20260812 11:15:44.851392 8405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } } }
I20260812 11:15:44.851545 8405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:44.851796 8405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4053ba429d4c4a881aeb0af8941bd1. Latest consensus state: current_term: 1 leader_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } } }
I20260812 11:15:44.851890 8405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:44.852232 8425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:44.857086 8425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:44.865542 8425 catalog_manager.cc:1347] Generated new cluster ID: e21f3bbc413a4b4ebbec38b6358120be
I20260812 11:15:44.865689 8425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:44.878960 8425 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:44.880633 8425 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:44.894477 8425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Generated new TSK 0
I20260812 11:15:44.896287 8425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:15:44.953197 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36075
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:45.115113 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:45.115612 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:45.115700 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:45.120757 8443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:45.120944 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:45.121076 8443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:45.127990 8443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:45.129693 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:45.131309 8443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:45.145130 8459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:45.145130 8452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:45.145131 8456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:45.145653 8443 server_base.cc:1034] running on GCE node
I20260812 11:15:45.146522 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:45.147570 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:45.148867 8443 hybrid_clock.cc:648] HybridClock initialized: now 1786533345148817 us; error 80 us; skew 500 ppm
Aug 12 11:15:45 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:45.152890 8443 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:15:45.154150 8443 webserver.cc:460] Webserver started at http://127.27.109.1:45019/ using document root <none> and password file <none>
I20260812 11:15:45.154928 8443 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:45.155022 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:45.155246 8443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:45.157714 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "4662b28a5b2c47f98a44ef799934c886"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "170d934ca433e7f7ee0a23ccdbdbd736"
server_key_iv: "819729e1ee64e76ce146678a36c2cc81"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.158628 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "4662b28a5b2c47f98a44ef799934c886"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "170d934ca433e7f7ee0a23ccdbdbd736"
server_key_iv: "819729e1ee64e76ce146678a36c2cc81"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.163913 8443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:15:45.167690 8467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.169770 8443 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:45.170004 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "4662b28a5b2c47f98a44ef799934c886"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "170d934ca433e7f7ee0a23ccdbdbd736"
server_key_iv: "819729e1ee64e76ce146678a36c2cc81"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.170171 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:45.187711 8443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:45.192266 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:45.192538 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:45.193275 8443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:45.194386 8443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:45.194479 8443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.194557 8443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:45.194607 8443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.207235 8443 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:43677
I20260812 11:15:45.207258 8643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:43677 every 8 connection(s)
I20260812 11:15:45.208395 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:45.217736 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 8443
I20260812 11:15:45.217940 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:45.218415 28084 external_mini_cluster.cc:1428] Setting key 3d27b9668e19cdddc42009e6f1f1fd1c
Aug 12 11:15:45 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:45.233080 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.211375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43657 (local address 127.27.109.62:36075)
0812 11:15:45.211613 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:45.211620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:45.212835 (+ 1215us) server_negotiation.cc:407] Connection header received
0812 11:15:45.214676 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:45.214681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:45.214754 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:45.215012 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:45.217957 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.218864 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.220724 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.221075 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.221204 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.226347 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:45.226351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:45.226353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:45.226386 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:45.228831 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.229712 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.229721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.229722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.229860 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.230921 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.230929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.230932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.231223 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:45.231345 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:45.232698 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0812 11:15:45.232892 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 11:15:45.234956 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.211877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:43657)
0812 11:15:45.212574 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:45.212644 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:45.214271 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:45.215436 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:45.215448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:45.216229 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:45.217617 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:45.217652 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.219121 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.219126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:45.220521 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:45.220540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.221339 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.221344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:45.221647 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.222804 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:45.222912 (+ 108us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:45.226036 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:45.229084 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:45.229093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:45.229117 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:45.229557 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:45.230117 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:45.230127 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:45.230131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:45.230589 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:45.231425 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:45.231441 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:45.232137 (+ 696us) client_negotiation.cc:769] Sending connection context
0812 11:15:45.232545 (+ 408us) client_negotiation.cc:240] Negotiation successful
0812 11:15:45.233236 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":162,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:15:45.237190 8644 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36075
I20260812 11:15:45.237571 8644 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:45.238490 8644 heartbeater.cc:511] Master 127.27.109.62:36075 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:15:45.241176 8328 ts_manager.cc:194] Registered new tserver with Master: 4662b28a5b2c47f98a44ef799934c886 (127.27.109.1:43677)
I20260812 11:15:45.242944 8328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:43657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:15:45.264714 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36075
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:45.430850 8654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:45.431380 8654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:45.431439 8654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:45.436167 8654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:45.436264 8654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:45.436335 8654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:45.442862 8654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:45.444416 8654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:45.445767 8654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:45.459946 8665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:45.459885 8660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:45.459965 8659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:45.460554 8654 server_base.cc:1034] running on GCE node
I20260812 11:15:45.461158 8654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:45.462002 8654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:45.463235 8654 hybrid_clock.cc:648] HybridClock initialized: now 1786533345463214 us; error 51 us; skew 500 ppm
Aug 12 11:15:45 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:45.466903 8654 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:15:45.468120 8654 webserver.cc:460] Webserver started at http://127.27.109.2:38313/ using document root <none> and password file <none>
I20260812 11:15:45.468829 8654 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:45.468935 8654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:45.469172 8654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:45.471599 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "2bffd58078a54587941adda2fa809a17"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "46fc09f4880682fe41d4db48d219dfa4"
server_key_iv: "18d641b8e2716628043a69125c3f9faf"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.472195 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "2bffd58078a54587941adda2fa809a17"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "46fc09f4880682fe41d4db48d219dfa4"
server_key_iv: "18d641b8e2716628043a69125c3f9faf"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.477178 8654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:45.480922 8678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.483059 8654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:15:45.483244 8654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "2bffd58078a54587941adda2fa809a17"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "46fc09f4880682fe41d4db48d219dfa4"
server_key_iv: "18d641b8e2716628043a69125c3f9faf"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.483531 8654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:45.504040 8654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:45.508466 8654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:45.508723 8654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:45.509544 8654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:45.510677 8654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:45.510771 8654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.510872 8654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:45.510939 8654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 11:15:45.524331 8654 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43609
I20260812 11:15:45.524381 8874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43609 every 8 connection(s)
I20260812 11:15:45.525521 8654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:45.528095 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 8654
I20260812 11:15:45.528295 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:45.528636 28084 external_mini_cluster.cc:1428] Setting key 6cd623dea22ca8d46bfef162f833f58e
Aug 12 11:15:45 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:15:45.547219 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.528443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:51133 (local address 127.27.109.62:36075)
0812 11:15:45.528605 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:45.528609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:45.530171 (+ 1562us) server_negotiation.cc:407] Connection header received
0812 11:15:45.532602 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:45.532606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:45.532690 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:45.532795 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:45.534603 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.535255 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.537448 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.537806 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.537992 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.541291 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:45.541300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:45.541304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:45.541366 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:45.543710 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.544640 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.544645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.544647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.544797 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.545340 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.545343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.545345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.545651 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:45.545842 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:45.546711 (+ 869us) server_negotiation.cc:299] Negotiation successful
0812 11:15:45.547011 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:15:45.548301 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.528962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:51133)
0812 11:15:45.529868 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:45.529962 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:45.532212 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:45.533096 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:45.533110 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:45.533731 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:45.534364 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:45.534383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.535423 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.535426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:45.536890 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:45.536906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.538056 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.538060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:45.538287 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.539066 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:45.539085 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:45.540916 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:45.543945 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:45.543953 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:45.543975 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:45.544439 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:45.544962 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:45.544966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:45.544968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:45.545175 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:45.545873 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:45.545890 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:45.546274 (+ 384us) client_negotiation.cc:769] Sending connection context
0812 11:15:45.546567 (+ 293us) client_negotiation.cc:240] Negotiation successful
0812 11:15:45.547016 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":222,"threads_started":1}
I20260812 11:15:45.550562 8875 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36075
I20260812 11:15:45.551079 8875 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:45.551883 8875 heartbeater.cc:511] Master 127.27.109.62:36075 requested a full tablet report, sending...
I20260812 11:15:45.553550 8328 ts_manager.cc:194] Registered new tserver with Master: 2bffd58078a54587941adda2fa809a17 (127.27.109.2:43609)
I20260812 11:15:45.554500 8328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:51133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:15:45.576968 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36075
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:45.726526 8881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:45.727157 8881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:45.727227 8881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:45.732946 8881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:45.733073 8881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:45.733160 8881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:45.741178 8881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:45.743000 8881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:45.744768 8881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:45.760419 8895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:45.760360 8890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:45.760378 8886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:45.761041 8881 server_base.cc:1034] running on GCE node
I20260812 11:15:45.761677 8881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:45.762672 8881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:45.763949 8881 hybrid_clock.cc:648] HybridClock initialized: now 1786533345763925 us; error 43 us; skew 500 ppm
Aug 12 11:15:45 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:45.768779 8881 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:15:45.770498 8881 webserver.cc:460] Webserver started at http://127.27.109.3:46379/ using document root <none> and password file <none>
I20260812 11:15:45.771420 8881 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:45.771543 8881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:45.771865 8881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:45.774816 8881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "b902f7639ef54203a778b41dc905204f"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "28bb0fb51d82c86478786d4f63340264"
server_key_iv: "d7ac441bd1baa456941f91304b566472"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.775754 8881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "b902f7639ef54203a778b41dc905204f"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "28bb0fb51d82c86478786d4f63340264"
server_key_iv: "d7ac441bd1baa456941f91304b566472"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.781414 8881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:45.785672 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.788038 8881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:45.788257 8881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "b902f7639ef54203a778b41dc905204f"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "28bb0fb51d82c86478786d4f63340264"
server_key_iv: "d7ac441bd1baa456941f91304b566472"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.788480 8881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:45.813581 8881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:45.819744 8881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:45.820098 8881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:45.821202 8881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:45.822780 8881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:45.822898 8881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 11:15:45.822968 8881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:45.823004 8881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.839648 8881 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35957
I20260812 11:15:45.839811 9092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35957 every 8 connection(s)
I20260812 11:15:45.841089 8881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:45.850366 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 8881
I20260812 11:15:45.850530 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:45.850910 28084 external_mini_cluster.cc:1428] Setting key 0291259f37a8e24e52524765491e284e
Aug 12 11:15:45 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:15:45.862368 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.844161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37153 (local address 127.27.109.62:36075)
0812 11:15:45.844354 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:45.844359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:45.846121 (+ 1762us) server_negotiation.cc:407] Connection header received
0812 11:15:45.847941 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:45.847946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:45.848022 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:45.848144 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:45.849927 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.850800 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.853296 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.853680 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.853896 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.857637 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:45.857640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:45.857644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:45.857680 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:45.859399 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.860263 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.860268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.860270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.860383 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.860943 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.860946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.860947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.861176 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:45.861279 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:45.862084 (+ 805us) server_negotiation.cc:299] Negotiation successful
0812 11:15:45.862219 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:15:45.863803 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.844842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:37153)
0812 11:15:45.845741 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:45.845838 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:45.847621 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:45.848388 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:45.848396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:45.848868 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:45.849672 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:45.849690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.851144 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.851150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:45.853034 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:45.853069 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.854006 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.854022 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:45.854395 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.855265 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:45.855287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:45.857364 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:45.859618 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:45.859626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:45.859652 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:45.860118 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:45.860527 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:45.860530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:45.860532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:45.860786 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:45.861313 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:45.861321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:45.861775 (+ 454us) client_negotiation.cc:769] Sending connection context
0812 11:15:45.862022 (+ 247us) client_negotiation.cc:240] Negotiation successful
0812 11:15:45.862498 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":220,"threads_started":1}
I20260812 11:15:45.865867 9093 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36075
I20260812 11:15:45.866298 9093 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:45.867200 9093 heartbeater.cc:511] Master 127.27.109.62:36075 requested a full tablet report, sending...
I20260812 11:15:45.868827 8328 ts_manager.cc:194] Registered new tserver with Master: b902f7639ef54203a778b41dc905204f (127.27.109.3:35957)
I20260812 11:15:45.869549 8328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:37153
I20260812 11:15:45.881099 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
WARNING: no policy specified for oryx/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
WARNING: no policy specified for oryx/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
I20260812 11:15:46.001731 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 8443
I20260812 11:15:46.033862 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 8654
I20260812 11:15:46.052724 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 8881
I20260812 11:15:46.072196 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 8277
I20260812 11:15:46.087831 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36075
--webserver_interface=127.27.109.62
--webserver_port=39037
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:46.222642 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:46.223212 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:46.223325 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:46.227926 9105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:46.228063 9105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:46.228099 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:46.228142 9105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:46.228201 9105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:46.235342 9105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:46.236701 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:46.237933 9105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:46.248688 9120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:46.248699 9126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:46.248733 9116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:46.250111 9105 server_base.cc:1034] running on GCE node
I20260812 11:15:46.251144 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:46.252877 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:46.254123 9105 hybrid_clock.cc:648] HybridClock initialized: now 1786533346254111 us; error 98 us; skew 500 ppm
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533346, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:46.258637 9105 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:15:46.260162 9105 webserver.cc:460] Webserver started at http://127.27.109.62:39037/ using document root <none> and password file <none>
I20260812 11:15:46.261014 9105 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:46.261132 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:46.267187 9105 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:46.271045 9140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.273365 9105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:46.273768 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0f4053ba429d4c4a881aeb0af8941bd1"
format_stamp: "Formatted at 2026-08-12 11:15:44 on dist-test-slave-7z32"
server_key: "540e6c7d1530e4348cf3808e3937b066"
server_key_iv: "d908a093255aec0b7bd860999cb1b068"
server_key_version: "encryptionkey@0"
I20260812 11:15:46.275461 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:46.292093 9105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:46.296823 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:46.297132 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:46.308367 9105 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:36075
I20260812 11:15:46.308504 9223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:36075 every 8 connection(s)
I20260812 11:15:46.310272 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:46.316432 9224 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:15:46.318665 9224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Bootstrap starting.
I20260812 11:15:46.319890 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 9105
I20260812 11:15:46.320590 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:43677
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:36075
--webserver_port=45019
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:46.340209 9224 log.cc:824] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:46.348557 9224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:15:46.349082 9224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Bootstrap complete.
I20260812 11:15:46.354553 9224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:46.355221 9224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4053ba429d4c4a881aeb0af8941bd1, State: Initialized, Role: FOLLOWER
I20260812 11:15:46.355844 9224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:46.356024 9224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:46.356132 9224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:46.356263 9224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:15:46.358774 9224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:46.359337 9224 leader_election.cc:302] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f4053ba429d4c4a881aeb0af8941bd1; no voters:
I20260812 11:15:46.359809 9224 leader_election.cc:288] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:15:46.359936 9229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:15:46.360138 9229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [term 2 LEADER]: Becoming Leader. State: Replica: 0f4053ba429d4c4a881aeb0af8941bd1, State: Running, Role: LEADER
I20260812 11:15:46.360510 9229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } }
I20260812 11:15:46.361500 9224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:46.361898 9231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4053ba429d4c4a881aeb0af8941bd1. Latest consensus state: current_term: 2 leader_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } } }
I20260812 11:15:46.362172 9231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:46.362337 9234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4053ba429d4c4a881aeb0af8941bd1" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36075 } } }
I20260812 11:15:46.362485 9234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:46.362908 9236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:46.367062 9236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:46.367974 9236 catalog_manager.cc:1259] Loaded cluster ID: e21f3bbc413a4b4ebbec38b6358120be
I20260812 11:15:46.368089 9236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:46.370914 9236 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:46.371755 9236 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f4053ba429d4c4a881aeb0af8941bd1: Loaded TSK: 0
I20260812 11:15:46.373306 9236 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:15:46.502025 9226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:46.502640 9226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:46.502758 9226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:46.508139 9226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:46.508312 9226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:46.508440 9226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:15:46.518013 9226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:43677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=45019
--tserver_master_addrs=127.27.109.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:46.519980 9226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:46.521538 9226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:46.535315 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:46.535275 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:46.535589 9226 server_base.cc:1034] running on GCE node
W20260812 11:15:46.535315 9263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:46.536379 9226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:46.537271 9226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:46.538652 9226 hybrid_clock.cc:648] HybridClock initialized: now 1786533346538601 us; error 84 us; skew 500 ppm
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533346, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:46.542550 9226 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260812 11:15:46.543871 9226 webserver.cc:460] Webserver started at http://127.27.109.1:45019/ using document root <none> and password file <none>
I20260812 11:15:46.544634 9226 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:46.544778 9226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:46.550222 9226 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 11:15:46.553577 9278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.555830 9226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:15:46.556062 9226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "4662b28a5b2c47f98a44ef799934c886"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "170d934ca433e7f7ee0a23ccdbdbd736"
server_key_iv: "819729e1ee64e76ce146678a36c2cc81"
server_key_version: "encryptionkey@0"
I20260812 11:15:46.556797 9226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:46.578039 9226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:46.582480 9226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:46.582777 9226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:46.583583 9226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:46.584672 9226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:46.584757 9226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.584852 9226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:46.584904 9226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.598093 9226 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:43677
I20260812 11:15:46.598224 9478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:43677 every 8 connection(s)
I20260812 11:15:46.599574 9226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:46.606637 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 9226
I20260812 11:15:46.607456 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:43609
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:36075
--webserver_port=38313
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:46.622931 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:46.603138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:40043)
0812 11:15:46.603849 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:46.603917 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:46.605472 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:46.607328 (+ 1856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:46.607343 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:46.608041 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:46.609073 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.609110 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.610974 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.610982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.612627 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.612649 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.614164 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.614170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.614423 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.615695 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:46.615724 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:46.621158 (+ 5434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":153,"threads_started":1}
W20260812 11:15:46.623953 9479 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:36075 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.62:36075: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
W20260812 11:15:46.624007 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:46.603134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:40043 (local address 127.27.109.62:36075)
0812 11:15:46.603813 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:46.603845 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:46.604090 (+ 245us) server_negotiation.cc:407] Connection header received
0812 11:15:46.606056 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:46.606070 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:46.606469 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:46.607002 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:46.609409 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.610670 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.612862 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.613843 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.614094 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.621735 (+ 7641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:40043: BlockingRecv error: recv got EOF from 127.27.109.1:40043 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":190,"threads_started":1}
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:46.632298 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:46.624780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:50351)
0812 11:15:46.625054 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:46.625078 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:46.625218 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:46.625574 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:46.625578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:46.625822 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:46.625961 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.625968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.626961 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.626967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.627654 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.627665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.628239 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.628244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.628375 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.629173 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:46.629189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:46.632094 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162}
W20260812 11:15:46.632525 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:46.624886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50351 (local address 127.27.109.62:36075)
0812 11:15:46.625045 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:46.625213 (+ 168us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:46.625236 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:15:46.625355 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:46.625364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:46.625425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:46.625540 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:46.626122 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.626770 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.627811 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.628089 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.628216 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.632361 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:50351: BlockingRecv error: recv got EOF from 127.27.109.1:50351 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:46.640537 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:46.633362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:42225)
0812 11:15:46.633858 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:46.633882 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:46.634028 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:46.634358 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:46.634361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:46.634591 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:46.634715 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.634721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.635762 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.635766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.636379 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.636389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.636910 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.636914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.637001 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.637690 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:46.637702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:46.640321 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260812 11:15:46.640879 9479 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:15:46.641016 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:46.633430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:42225 (local address 127.27.109.62:36075)
0812 11:15:46.633862 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:46.633869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:46.633941 (+ 72us) server_negotiation.cc:407] Connection header received
0812 11:15:46.634166 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:46.634170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:46.634229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:46.634332 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:46.634933 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.635620 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.636538 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.636776 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.636896 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.640523 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:42225: BlockingRecv error: recv got EOF from 127.27.109.1:42225 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 11:15:46.748747 9487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:46.749121 9487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:46.749207 9487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:46.754339 9487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:46.754483 9487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:46.754586 9487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:15:46.760502 9487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:43609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=38313
--tserver_master_addrs=127.27.109.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:46.761955 9487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:46.763437 9487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:46.775475 9493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:46.775488 9494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:46.775516 9497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:46.776094 9487 server_base.cc:1034] running on GCE node
I20260812 11:15:46.776549 9487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:46.777318 9487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:46.778616 9487 hybrid_clock.cc:648] HybridClock initialized: now 1786533346778591 us; error 55 us; skew 500 ppm
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533346, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:46.782035 9487 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260812 11:15:46.783341 9487 webserver.cc:460] Webserver started at http://127.27.109.2:38313/ using document root <none> and password file <none>
I20260812 11:15:46.783959 9487 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:46.784049 9487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:46.788736 9487 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:15:46.792196 9509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.794366 9487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:46.794605 9487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "2bffd58078a54587941adda2fa809a17"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "46fc09f4880682fe41d4db48d219dfa4"
server_key_iv: "18d641b8e2716628043a69125c3f9faf"
server_key_version: "encryptionkey@0"
I20260812 11:15:46.795450 9487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:46.818776 9487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:46.824102 9487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:46.824456 9487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:46.825227 9487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:46.826273 9487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:46.826365 9487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.826447 9487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:46.826504 9487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.845800 9487 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43609
I20260812 11:15:46.845911 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43609 every 8 connection(s)
I20260812 11:15:46.847219 9487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:46.853433 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 9487
I20260812 11:15:46.854226 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:35957
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:36075
--webserver_port=46379
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:15:46.870344 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:46.851485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:42227 (local address 127.27.109.62:36075)
0812 11:15:46.852021 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:46.852026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:46.854267 (+ 2241us) server_negotiation.cc:407] Connection header received
0812 11:15:46.856424 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:46.856428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:46.856521 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:46.856651 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:46.859552 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.860387 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.862546 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.863144 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.863310 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.870120 (+ 6810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:42227: BlockingRecv error: recv got EOF from 127.27.109.2:42227 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 11:15:46.871738 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:46.852165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:42227)
0812 11:15:46.853911 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:46.854035 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:46.855943 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:46.857100 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:46.857121 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:46.858006 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:46.859214 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.859251 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.860632 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.860637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.862307 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.862329 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.863399 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.863404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.863706 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.865491 (+ 1785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:46.865522 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:46.869781 (+ 4259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1295,"thread_start_us":923,"threads_started":1}
W20260812 11:15:46.872830 9712 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:36075 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.62:36075: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:46.883754 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:46.874161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:40223)
0812 11:15:46.874788 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:46.874811 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:46.875231 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:46.875716 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:46.875719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:46.875959 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:46.876193 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.876202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.877470 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.877475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.878272 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.878289 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.879098 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.879101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.879226 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.880045 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:46.880064 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:46.883516 (+ 3452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315}
W20260812 11:15:46.883957 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:46.874265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:40223 (local address 127.27.109.62:36075)
0812 11:15:46.874455 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:46.874460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:46.875170 (+ 710us) server_negotiation.cc:407] Connection header received
0812 11:15:46.875407 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:46.875411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:46.875492 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:46.875636 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:46.876356 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.877262 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.878471 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.878899 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.879039 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.883799 (+ 4760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:40223: BlockingRecv error: recv got EOF from 127.27.109.2:40223 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:46 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:46.893383 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:46.884849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:34143)
0812 11:15:46.885017 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:46.885039 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:46.885163 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:46.885585 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:46.885589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:46.885834 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:46.885976 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.885984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.887209 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.887213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.888014 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:46.888034 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.889015 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.889019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:46.889124 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.889934 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:46.889951 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:46.893199 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 11:15:46.893718 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:46.884996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34143 (local address 127.27.109.62:36075)
0812 11:15:46.885124 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:46.885128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:46.885147 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:46.885289 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:46.885293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:46.885356 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:46.885468 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:46.886140 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.886990 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.888212 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:46.888566 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:46.889002 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:46.893520 (+ 4518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:34143: BlockingRecv error: recv got EOF from 127.27.109.2:34143 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 11:15:46.893824 9712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:15:47.006464 9717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:47.006780 9717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:47.006896 9717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:47.012071 9717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:47.012190 9717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:47.012264 9717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:15:47.019316 9717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:35957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=46379
--tserver_master_addrs=127.27.109.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:15:47.020577 9717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:47.021750 9717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:47.036017 9723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:47.036017 9726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:47.036046 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:47.036794 9717 server_base.cc:1034] running on GCE node
I20260812 11:15:47.037547 9717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:47.038612 9717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:47.042063 9717 hybrid_clock.cc:648] HybridClock initialized: now 1786533347041975 us; error 95 us; skew 500 ppm
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:47.047196 9717 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260812 11:15:47.048733 9717 webserver.cc:460] Webserver started at http://127.27.109.3:46379/ using document root <none> and password file <none>
I20260812 11:15:47.049621 9717 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:47.049791 9717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:47.056542 9717 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:47.060266 9736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:47.062951 9717 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:47.063191 9717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "b902f7639ef54203a778b41dc905204f"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-7z32"
server_key: "28bb0fb51d82c86478786d4f63340264"
server_key_iv: "d7ac441bd1baa456941f91304b566472"
server_key_version: "encryptionkey@0"
I20260812 11:15:47.064165 9717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:47.084038 9717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:47.088887 9717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:47.089210 9717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:47.090104 9717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:47.091485 9717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:47.091571 9717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:47.091661 9717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:47.091713 9717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:47.106748 9717 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35957
I20260812 11:15:47.106910 9927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35957 every 8 connection(s)
I20260812 11:15:47.108347 9717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:47.117043 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 9717
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:15:47.126725 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:47.111186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:42797 (local address 127.27.109.62:36075)
0812 11:15:47.111429 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.111439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.112845 (+ 1406us) server_negotiation.cc:407] Connection header received
0812 11:15:47.114575 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.114579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.114644 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.114766 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.116909 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.117651 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.120285 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.120699 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.120877 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.126512 (+ 5635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:42797: BlockingRecv error: recv got EOF from 127.27.109.3:42797 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 11:15:47.127673 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.111538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:42797)
0812 11:15:47.112549 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:47.112645 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:47.114252 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:47.115163 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:47.115172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:47.115796 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:47.116608 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.116630 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.117868 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.117872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.120047 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.120075 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.121049 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.121053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.121296 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.122222 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:47.122251 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:47.126237 (+ 3986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":220,"threads_started":1}
W20260812 11:15:47.128683 9928 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:36075 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.62:36075: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
I20260812 11:15:47.132874 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.118757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.27.109.62:36075)
0812 11:15:47.119137 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.119141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.119158 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:15:47.119215 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.119218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.119265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.119383 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.120722 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.121478 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.122959 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.123282 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.123436 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.125281 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:47.125290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:47.125320 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:47.125382 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:47.128283 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:47.129292 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:47.129304 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:47.129315 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:47.129594 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:47.130444 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:47.130450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:47.130456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:47.131143 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:47.131341 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:47.131884 (+ 543us) server_negotiation.cc:299] Negotiation successful
0812 11:15:47.132544 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":213,"threads_started":1}
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:47.137571 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.129520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:35349)
0812 11:15:47.129805 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:47.129824 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:47.129950 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:47.130317 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:47.130321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:47.130577 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:47.130736 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.130746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.132203 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.132208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.133300 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.133318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.133889 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.133893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.134008 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.134648 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:47.134664 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:47.137406 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179}
W20260812 11:15:47.138137 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:47.129674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:35349 (local address 127.27.109.62:36075)
0812 11:15:47.129880 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.129885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.129900 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:15:47.130115 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.130118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.130178 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.130287 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.130999 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.132015 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.133474 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.133738 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.133858 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.137609 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:35349: BlockingRecv error: recv got EOF from 127.27.109.3:35349 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:47.145136 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.138412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:58633)
0812 11:15:47.138664 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:47.138677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:47.138758 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:47.139174 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:47.139188 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:47.139420 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:47.139525 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.139531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.140461 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.140464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.141059 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.141069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.141653 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.141656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.141726 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.142336 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:47.142349 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:47.145013 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180}
W20260812 11:15:47.145488 9928 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:15:47.145490 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:47.138496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:58633 (local address 127.27.109.62:36075)
0812 11:15:47.138722 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.138725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.138742 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:15:47.138927 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.138930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.138996 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.139096 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.139697 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.140304 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.141213 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.141513 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.141619 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.145319 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:58633: BlockingRecv error: recv got EOF from 127.27.109.3:58633 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:47.650729 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.641967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:51591)
0812 11:15:47.642464 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:47.642490 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:47.642727 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:47.643167 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:47.643171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:47.643469 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:47.643615 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.643624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.644573 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.644579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.645429 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.645442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.646008 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.646013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.646107 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.646913 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:47.646935 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:47.650563 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":174,"threads_started":1}
W20260812 11:15:47.651223 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:47.642230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:51591 (local address 127.27.109.62:36075)
0812 11:15:47.642464 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.642473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.642628 (+ 155us) server_negotiation.cc:407] Connection header received
0812 11:15:47.642911 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.642915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.642993 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.643127 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.643773 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.644396 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.645576 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.645840 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.645986 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.651048 (+ 5062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:51591: BlockingRecv error: recv got EOF from 127.27.109.1:51591 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:47 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:47.902375 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.895083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:53037)
0812 11:15:47.895527 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:47.895553 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:47.895769 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:47.896176 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:47.896179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:47.896441 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:47.896561 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.896569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.897484 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.897492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.898168 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:47.898178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.898728 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.898732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:47.898888 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.899660 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:47.899675 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:47.902137 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":131,"threads_started":1}
W20260812 11:15:47.902515 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:47.895140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:53037 (local address 127.27.109.62:36075)
0812 11:15:47.895503 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.895510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.895681 (+ 171us) server_negotiation.cc:407] Connection header received
0812 11:15:47.895964 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.895967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.896025 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.896137 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.896712 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.897346 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.898325 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.898571 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.898701 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.902354 (+ 3653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:53037: BlockingRecv error: recv got EOF from 127.27.109.2:53037 (error 108)
Metrics: {"server-negotiator.queue_time_us":214}
Aug 12 11:15:48 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:48 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:48.161239 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:48.146655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:49571)
0812 11:15:48.147110 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:48.147133 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:48.147390 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:48.147716 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:48.147719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:48.148116 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:48.148266 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:48.148274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.149314 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.149320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:48.150403 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:48.150419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.151135 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.151141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:48.151293 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:48.158767 (+ 7474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:48.158790 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:48.161013 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":150,"threads_started":1}
W20260812 11:15:48.161327 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:48.146866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:49571 (local address 127.27.109.62:36075)
0812 11:15:48.147089 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:48.147095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:48.147248 (+ 153us) server_negotiation.cc:407] Connection header received
0812 11:15:48.147520 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:48.147525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:48.147582 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:48.147681 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:48.148451 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.149085 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.150615 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.150964 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.151102 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:48.161207 (+ 10105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:49571: BlockingRecv error: recv got EOF from 127.27.109.3:49571 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 11:15:48 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:48 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:48.661739 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:48.652157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:36455)
0812 11:15:48.652566 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:48.652775 (+ 209us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:48.653005 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:48.653576 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:48.653579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:48.653823 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:48.653982 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:48.653990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.655534 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.655540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:48.656335 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:48.656354 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.656966 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.656971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:48.657108 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:48.657906 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:48.657923 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:48.661309 (+ 3386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":171,"threads_started":1}
W20260812 11:15:48.661841 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:48.652357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:36455 (local address 127.27.109.62:36075)
0812 11:15:48.652689 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:48.652695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:48.652951 (+ 256us) server_negotiation.cc:407] Connection header received
0812 11:15:48.653223 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:48.653227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:48.653294 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:48.653414 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:48.654177 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.655325 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.656521 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.656801 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.656945 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:48.661638 (+ 4693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:36455: BlockingRecv error: recv got EOF from 127.27.109.1:36455 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 11:15:48 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:48 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:48.911249 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:48.903557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:56565)
0812 11:15:48.903886 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:48.904005 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:48.904179 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:48.904614 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:48.904618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:48.904889 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:48.905030 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:48.905037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.906113 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.906116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:48.906814 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:48.906871 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.907563 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.907566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:48.907654 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:48.908370 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:48.908391 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:48.911059 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":125,"threads_started":1}
W20260812 11:15:48.911614 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:48.903756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:56565 (local address 127.27.109.62:36075)
0812 11:15:48.903936 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:48.903941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:48.904135 (+ 194us) server_negotiation.cc:407] Connection header received
0812 11:15:48.904346 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:48.904350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:48.904417 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:48.904530 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:48.905241 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.905918 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.907011 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:48.907315 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:48.907435 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:48.911386 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:56565: BlockingRecv error: recv got EOF from 127.27.109.2:56565 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 11:15:49 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:49 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:49.170307 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:49.162724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:57761)
0812 11:15:49.163211 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:49.163234 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:49.163449 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:49.163792 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:49.163796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:49.164119 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:49.164266 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:49.164276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.165237 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.165242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:49.165880 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:49.165888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.166423 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.166427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:49.166539 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:49.167279 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:49.167294 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:49.170080 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":154,"threads_started":1}
W20260812 11:15:49.170537 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:49.163005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57761 (local address 127.27.109.62:36075)
0812 11:15:49.163211 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:49.163219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:49.163350 (+ 131us) server_negotiation.cc:407] Connection header received
0812 11:15:49.163591 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:49.163594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:49.163664 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:49.163761 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:49.164376 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.165103 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.166018 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.166279 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.166409 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:49.170340 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:57761: BlockingRecv error: recv got EOF from 127.27.109.3:57761 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 11:15:49 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:49 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:49.671739 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:49.663268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:46915)
0812 11:15:49.663583 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:49.663609 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:49.663735 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:49.664223 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:49.664226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:49.664512 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:49.664641 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:49.664647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.665655 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.665658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:49.666286 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:49.666296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.667063 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.667066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:49.667207 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:49.667887 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:49.667900 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:49.671590 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":115,"threads_started":1}
W20260812 11:15:49.672101 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:49.663442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:46915 (local address 127.27.109.62:36075)
0812 11:15:49.663679 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:49.663684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:49.663709 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:15:49.663898 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:49.663902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:49.663968 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:49.664087 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:49.664818 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.665452 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.666478 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.666772 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.667073 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:49.671941 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:46915: BlockingRecv error: recv got EOF from 127.27.109.1:46915 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 11:15:49 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:49 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:49.928122 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:49.912389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:44063)
0812 11:15:49.913118 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:49.913138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:49.913453 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:49.914034 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:49.914039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:49.914430 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:49.914612 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:49.914621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.919213 (+ 4592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.919218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:49.920384 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:49.920406 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.921613 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.921617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:49.921707 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:49.923230 (+ 1523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:49.923249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:49.927793 (+ 4544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":119,"threads_started":1}
W20260812 11:15:49.928418 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:49.912660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:44063 (local address 127.27.109.62:36075)
0812 11:15:49.912824 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:49.912831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:49.913219 (+ 388us) server_negotiation.cc:407] Connection header received
0812 11:15:49.913684 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:49.913688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:49.913773 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:49.913911 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:49.915080 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.918471 (+ 3391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.921173 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:49.921460 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:49.921596 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:49.928079 (+ 6483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:44063: BlockingRecv error: recv got EOF from 127.27.109.2:44063 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 11:15:50 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:50 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:50.179160 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:50.171453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:42355)
0812 11:15:50.171909 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:50.171931 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:50.172228 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:50.172636 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:50.172639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:50.172875 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:50.173010 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:50.173017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.174080 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.174083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:50.174799 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:50.174814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.175591 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.175595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:50.175680 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.176578 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:50.176592 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:50.179005 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":154,"threads_started":1}
W20260812 11:15:50.179663 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:50.171770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:42355 (local address 127.27.109.62:36075)
0812 11:15:50.171939 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:50.171945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:50.172210 (+ 265us) server_negotiation.cc:407] Connection header received
0812 11:15:50.172378 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:50.172380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:50.172449 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:50.172538 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:50.173194 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.173853 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.175141 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.175398 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.175510 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.179427 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:42355: BlockingRecv error: recv got EOF from 127.27.109.3:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 11:15:50 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:50 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:50.679209 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:50.672963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:33279)
0812 11:15:50.673393 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:50.673420 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:50.673625 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:50.673929 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:50.673933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:50.674190 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:50.674289 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:50.674295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.675164 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.675168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:50.675754 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:50.675763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.676257 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.676261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:50.676349 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.676949 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:50.676968 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:50.679009 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":183,"threads_started":1}
W20260812 11:15:50.679291 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:50.673098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33279 (local address 127.27.109.62:36075)
0812 11:15:50.673363 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:50.673369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:50.673548 (+ 179us) server_negotiation.cc:407] Connection header received
0812 11:15:50.673747 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:50.673755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:50.673805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:50.673902 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:50.674418 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.675048 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.675895 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.676116 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.676229 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.679158 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:33279: BlockingRecv error: recv got EOF from 127.27.109.1:33279 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Aug 12 11:15:50 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:50 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:50.938143 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:50.929525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:55483)
0812 11:15:50.930071 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:50.930095 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:50.930301 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:50.930634 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:50.930637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:50.930920 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:50.931036 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:50.931042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.932050 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.932056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:50.933193 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:50.933212 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.934101 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.934105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:50.934213 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.935054 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:50.935069 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:50.937946 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":138,"threads_started":1}
W20260812 11:15:50.938476 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:50.929772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:55483 (local address 127.27.109.62:36075)
0812 11:15:50.930039 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:50.930045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:50.930219 (+ 174us) server_negotiation.cc:407] Connection header received
0812 11:15:50.930426 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:50.930429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:50.930492 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:50.930598 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:50.931187 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.931784 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.933430 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.933814 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.934011 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.938204 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:55483: BlockingRecv error: recv got EOF from 127.27.109.2:55483 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 12 11:15:51 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:51 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:51.188524 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:51.180695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:45547)
0812 11:15:51.181136 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:51.181157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:51.181346 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:51.181780 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:51.181783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:51.182118 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:51.182267 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:51.182274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.183327 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.183333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:51.184071 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:51.184085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.185062 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.185068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:51.185238 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:51.186060 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:51.186084 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:51.188337 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":93,"threads_started":1}
W20260812 11:15:51.188993 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:51.180887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:45547 (local address 127.27.109.62:36075)
0812 11:15:51.181106 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:51.181110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:51.181307 (+ 197us) server_negotiation.cc:407] Connection header received
0812 11:15:51.181533 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:51.181546 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:51.181624 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:51.181741 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:51.182429 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.183187 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.184269 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.184709 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.184822 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:51.188795 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:45547: BlockingRecv error: recv got EOF from 127.27.109.3:45547 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 12 11:15:51 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:51 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:51.688284 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:51.680440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:57915)
0812 11:15:51.680998 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:51.681018 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:51.681242 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:51.681723 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:51.681727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:51.682036 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:51.682167 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:51.682174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.683226 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.683231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:51.683922 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:51.683932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.684505 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.684509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:51.684626 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:51.685383 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:51.685398 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:51.688068 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":114,"threads_started":1}
W20260812 11:15:51.688438 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:51.680606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57915 (local address 127.27.109.62:36075)
0812 11:15:51.680923 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:51.680931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:51.681183 (+ 252us) server_negotiation.cc:407] Connection header received
0812 11:15:51.681405 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:51.681409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:51.681478 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:51.681657 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:51.682318 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.683090 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.684081 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.684348 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.684486 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:51.688303 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:57915: BlockingRecv error: recv got EOF from 127.27.109.1:57915 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Aug 12 11:15:51 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:51 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:51.949146 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:51.939536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:39183)
0812 11:15:51.939955 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:51.939980 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:51.940206 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:51.940836 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:51.940841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:51.941189 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:51.941405 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:51.941415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.942515 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.942520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:51.943796 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:51.943835 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.945031 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.945039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:51.945162 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:51.945964 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:51.945986 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:51.948920 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":149,"threads_started":1}
W20260812 11:15:51.949282 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:51.939761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39183 (local address 127.27.109.62:36075)
0812 11:15:51.939948 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:51.939954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:51.940118 (+ 164us) server_negotiation.cc:407] Connection header received
0812 11:15:51.940358 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:51.940364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:51.940443 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:51.940552 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:51.941727 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.942353 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.944078 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:51.944576 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:51.944703 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:51.949143 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:39183: BlockingRecv error: recv got EOF from 127.27.109.2:39183 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 11:15:52 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:52 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:52.197557 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:52.189989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:53833)
0812 11:15:52.190314 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:52.190332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:52.190500 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:52.191050 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:52.191054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:52.191368 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:52.191497 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:52.191503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.192613 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.192616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:52.193356 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:52.193367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.193963 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.193973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:52.194070 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.194906 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:52.194940 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:52.197355 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":122,"threads_started":1}
W20260812 11:15:52.197854 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:52.190148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:53833 (local address 127.27.109.62:36075)
0812 11:15:52.190339 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:52.190346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:52.190424 (+ 78us) server_negotiation.cc:407] Connection header received
0812 11:15:52.190705 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:52.190712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:52.190789 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:52.191007 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:52.191818 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.192452 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.193504 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.193771 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.193878 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.197623 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:53833: BlockingRecv error: recv got EOF from 127.27.109.3:53833 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 11:15:52 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:52 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:52.696995 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:52.689569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:44305)
0812 11:15:52.689942 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:52.689963 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:52.690148 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:52.690553 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:52.690556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:52.690798 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:52.691049 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:52.691057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.692069 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.692072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:52.692705 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:52.692715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.693255 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.693258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:52.693329 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.694103 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:52.694117 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:52.696831 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
W20260812 11:15:52.697343 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:52.689746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44305 (local address 127.27.109.62:36075)
0812 11:15:52.689916 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:52.689921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:52.690069 (+ 148us) server_negotiation.cc:407] Connection header received
0812 11:15:52.690263 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:52.690267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:52.690330 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:52.690445 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:52.691221 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.691882 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.692862 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.693123 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.693227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.697160 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:44305: BlockingRecv error: recv got EOF from 127.27.109.1:44305 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 11:15:52 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:52 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:52.961299 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:52.951265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:50641)
0812 11:15:52.951894 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:52.951923 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:52.952136 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:52.952864 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:52.952868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:52.953145 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:52.953300 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:52.953308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.954441 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.954444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:52.955502 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:52.955525 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.956517 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.956521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:52.956622 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.957584 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:52.957603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:52.961074 (+ 3471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":104,"threads_started":1}
W20260812 11:15:52.961418 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:52.951527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:50641 (local address 127.27.109.62:36075)
0812 11:15:52.951769 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:52.951773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:52.952120 (+ 347us) server_negotiation.cc:407] Connection header received
0812 11:15:52.952313 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:52.952316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:52.952479 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:52.952618 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:52.953571 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.954218 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.955750 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.956274 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.956428 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.961272 (+ 4844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:50641: BlockingRecv error: recv got EOF from 127.27.109.2:50641 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 12 11:15:53 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:53 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:53.213397 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:53.198912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:36413)
0812 11:15:53.199413 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:53.199440 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:53.199649 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:53.200148 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:53.200152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:53.200442 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:53.200565 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:53.200572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.201946 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.201950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:53.202911 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:53.202924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.203503 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.203512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:53.203590 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:53.210569 (+ 6979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:53.210596 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:53.213182 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":168,"threads_started":1}
W20260812 11:15:53.213752 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:53.199218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:36413 (local address 127.27.109.62:36075)
0812 11:15:53.199402 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:53.199407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:53.199682 (+ 275us) server_negotiation.cc:407] Connection header received
0812 11:15:53.199789 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:53.199793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:53.199874 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:53.199996 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:53.200832 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.201697 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.203089 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.203350 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.203462 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:53.213560 (+ 10098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:36413: BlockingRecv error: recv got EOF from 127.27.109.3:36413 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 11:15:53 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:53 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:53.707000 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:53.698433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:57311)
0812 11:15:53.698922 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:53.698940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:53.699132 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:53.699499 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:53.699502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:53.699781 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:53.699899 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:53.699905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.700825 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.700829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:53.701451 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:53.701461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.702274 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.702279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:53.702389 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:53.703882 (+ 1493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:53.703922 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:53.706772 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":159,"threads_started":1}
W20260812 11:15:53.707372 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:53.698956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57311 (local address 127.27.109.62:36075)
0812 11:15:53.699189 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:53.699197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:53.699218 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:15:53.699299 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:53.699301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:53.699358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:53.699466 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:53.700036 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.700638 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.701597 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.702004 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.702139 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:53.707183 (+ 5044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:57311: BlockingRecv error: recv got EOF from 127.27.109.1:57311 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 12 11:15:53 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:53 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:53.971278 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:53.962578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:35895)
0812 11:15:53.963034 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:53.963051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:53.963229 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:53.963675 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:53.963677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:53.964124 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:53.964267 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:53.964276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.965665 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.965668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:53.966362 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:53.966374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.967198 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.967202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:53.967290 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:53.968000 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:53.968014 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:53.970963 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":114,"threads_started":1}
W20260812 11:15:53.971328 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:53.962679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:35895 (local address 127.27.109.62:36075)
0812 11:15:53.962983 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:53.962991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:53.963203 (+ 212us) server_negotiation.cc:407] Connection header received
0812 11:15:53.963408 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:53.963412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:53.963483 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:53.963602 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:53.964539 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.965402 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.966569 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:53.966995 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:53.967139 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:53.971182 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:35895: BlockingRecv error: recv got EOF from 127.27.109.2:35895 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 11:15:54 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:54 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:54.222859 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:54.215072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:40151)
0812 11:15:54.215481 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:54.215495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:54.215625 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:54.216021 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:54.216024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:54.216248 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:54.216429 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:54.216435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.217531 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.217534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:54.218160 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:54.218169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.219232 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.219235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:54.219327 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:54.220240 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:54.220257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:54.222626 (+ 2369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":148,"threads_started":1}
W20260812 11:15:54.223486 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:54.215268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:40151 (local address 127.27.109.62:36075)
0812 11:15:54.215483 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:54.215488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:54.215671 (+ 183us) server_negotiation.cc:407] Connection header received
0812 11:15:54.215748 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:54.215751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:54.215814 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:54.215940 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:54.216693 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.217355 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.218618 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.219024 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.219181 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:54.223338 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:40151: BlockingRecv error: recv got EOF from 127.27.109.3:40151 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 12 11:15:54 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:54 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:54.716848 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:54.708278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:44427)
0812 11:15:54.708734 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:54.708752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:54.708886 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:54.709345 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:54.709348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:54.709619 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:54.709747 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:54.709753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.710813 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.710816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:54.711967 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:54.711979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.712593 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.712596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:54.712673 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:54.713394 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:54.713408 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:54.716580 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":167,"threads_started":1}
W20260812 11:15:54.716984 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:54.708652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44427 (local address 127.27.109.62:36075)
0812 11:15:54.708807 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:54.708811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:54.708830 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:54.709098 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:54.709102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:54.709171 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:54.709278 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:54.709923 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.710597 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.712152 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.712428 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.712547 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:54.716788 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:44427: BlockingRecv error: recv got EOF from 127.27.109.1:44427 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 11:15:54 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:54 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:54.981173 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:54.972756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:60317)
0812 11:15:54.973210 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:54.973233 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:54.973544 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:54.974146 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:54.974149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:54.974428 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:54.974632 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:54.974641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.975942 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.975949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:54.976670 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:54.976683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.977205 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.977208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:54.977290 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:54.978133 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:54.978146 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:54.980966 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":93,"threads_started":1}
W20260812 11:15:54.981546 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:54.973015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:60317 (local address 127.27.109.62:36075)
0812 11:15:54.973254 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:54.973258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:54.973407 (+ 149us) server_negotiation.cc:407] Connection header received
0812 11:15:54.973770 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:54.973774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:54.973886 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:54.974003 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:54.974860 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.975764 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.976819 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:54.977061 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:54.977189 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:54.981354 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:60317: BlockingRecv error: recv got EOF from 127.27.109.2:60317 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 12 11:15:55 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:55 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:55.233150 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.224251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:49323)
0812 11:15:55.224856 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.224876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:55.225100 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:55.225504 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:55.225507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:55.225819 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:55.225948 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.225955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.227189 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.227196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.228147 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.228160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.229307 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.229311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.229511 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.230232 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:55.230246 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:55.233000 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":122,"threads_started":1}
W20260812 11:15:55.233639 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:55.224481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:49323 (local address 127.27.109.62:36075)
0812 11:15:55.224763 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:55.224772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:55.225005 (+ 233us) server_negotiation.cc:407] Connection header received
0812 11:15:55.225249 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:55.225254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:55.225341 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:55.225473 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:55.226104 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.226789 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.228483 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.229109 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.229252 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.233451 (+ 4199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:49323: BlockingRecv error: recv got EOF from 127.27.109.3:49323 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Aug 12 11:15:55 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:55 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:55.725804 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.718155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:60719)
0812 11:15:55.718552 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.718568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:55.718712 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:55.719084 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:55.719087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:55.719358 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:55.719493 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.719499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.720437 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.720440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.721633 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.721652 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.722425 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.722428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.722508 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.723231 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:55.723244 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:55.725599 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":156,"threads_started":1}
W20260812 11:15:55.726135 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:55.718267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:60719 (local address 127.27.109.62:36075)
0812 11:15:55.718481 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:55.718486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:55.718671 (+ 185us) server_negotiation.cc:407] Connection header received
0812 11:15:55.718859 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:55.718861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:55.718919 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:55.719036 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:55.719661 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.720250 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.721877 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.722267 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.722392 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.725916 (+ 3524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:60719: BlockingRecv error: recv got EOF from 127.27.109.1:60719 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 12 11:15:55 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:55 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:55.991245 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.982750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:36229)
0812 11:15:55.983252 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.983269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:55.983438 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:55.983954 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:55.983958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:55.984234 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:55.984375 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.984383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.985239 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.985245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.985822 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.985848 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.986391 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.986395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.986475 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.987542 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:55.987557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:55.991069 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":196,"threads_started":1}
W20260812 11:15:55.991735 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:55.982964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:36229 (local address 127.27.109.62:36075)
0812 11:15:55.983167 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:55.983173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:55.983370 (+ 197us) server_negotiation.cc:407] Connection header received
0812 11:15:55.983634 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:55.983638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:55.983719 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:55.983835 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:55.984549 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.985103 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.986010 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.986244 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.986372 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.991543 (+ 5171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:36229: BlockingRecv error: recv got EOF from 127.27.109.2:36229 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 12 11:15:56 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:56 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:56.243352 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.234559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:59311)
0812 11:15:56.234947 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.234968 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.235106 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.235559 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.235562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.235866 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.236046 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.236052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.237224 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.237227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.238336 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.238363 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.239326 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.239331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.239416 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.240251 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.240270 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.243187 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
W20260812 11:15:56.243774 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.235007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:59311 (local address 127.27.109.62:36075)
0812 11:15:56.235198 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.235203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.235229 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:15:56.235282 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.235284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.235373 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.235484 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.236214 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.237076 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.238610 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.239130 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.239290 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.243534 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:59311: BlockingRecv error: recv got EOF from 127.27.109.3:59311 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 11:15:56 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:56 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:56.734872 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.727320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:58239)
0812 11:15:56.727836 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.727858 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.728064 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.728423 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.728425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.728703 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.728825 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.728832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.729733 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.729738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.730698 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.730711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.731341 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.731345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.731440 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.732194 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.732219 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.734606 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":155,"threads_started":1}
W20260812 11:15:56.735018 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.727573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:58239 (local address 127.27.109.62:36075)
0812 11:15:56.727820 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.727826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.727984 (+ 158us) server_negotiation.cc:407] Connection header received
0812 11:15:56.728210 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.728215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.728276 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.728381 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.728958 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.729589 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.730911 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.731184 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.731324 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.734814 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:58239: BlockingRecv error: recv got EOF from 127.27.109.1:58239 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 11:15:56 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:56 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:56.998908 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.992381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:57865)
0812 11:15:56.992715 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.992740 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.992843 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.993242 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.993244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.993511 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.993629 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.993635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.994593 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.994596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.995279 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.995289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.995920 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.995924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.996001 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.996694 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.996707 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.998701 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":137,"threads_started":1}
W20260812 11:15:56.999321 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.992554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:57865 (local address 127.27.109.62:36075)
0812 11:15:56.992756 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.992761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.992790 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:15:56.992973 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.992976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.993074 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.993207 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.993795 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.994441 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.995448 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.995713 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.995837 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.999139 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:57865: BlockingRecv error: recv got EOF from 127.27.109.2:57865 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 11:15:57 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:57 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:57.251739 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:57.244896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:50889)
0812 11:15:57.245469 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:57.245493 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:57.245695 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:57.246024 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:57.246027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:57.246259 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:57.246361 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:57.246367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.247253 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.247257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:57.247832 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:57.247840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.248329 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.248334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:57.248413 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.249038 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:57.249051 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:57.251576 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":161,"threads_started":1}
W20260812 11:15:57.252036 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:57.245236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:50889 (local address 127.27.109.62:36075)
0812 11:15:57.245488 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:57.245495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:57.245609 (+ 114us) server_negotiation.cc:407] Connection header received
0812 11:15:57.245831 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:57.245836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:57.245891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:57.245989 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:57.246486 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.247128 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.247961 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.248194 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.248321 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.251778 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:50889: BlockingRecv error: recv got EOF from 127.27.109.3:50889 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Aug 12 11:15:57 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:57 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:57.744635 9986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:57.736299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:55435)
0812 11:15:57.736759 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:57.736777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:57.737028 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:57.737472 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:57.737475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:57.737859 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:57.738118 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:57.738128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.739113 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.739117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:57.739876 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:57.739890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.740605 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.740608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:57.740711 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.741775 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:57.741791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:57.744455 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":201,"threads_started":1}
W20260812 11:15:57.744869 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:57.736391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:55435 (local address 127.27.109.62:36075)
0812 11:15:57.736649 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:57.736655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:57.737073 (+ 418us) server_negotiation.cc:407] Connection header received
0812 11:15:57.737153 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:57.737157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:57.737225 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:57.737341 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:57.738305 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.738909 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.740152 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.740433 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.740550 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.744699 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:55435: BlockingRecv error: recv got EOF from 127.27.109.1:55435 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Aug 12 11:15:58 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:58 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:58.007007 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:58.000038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:56237)
0812 11:15:58.000627 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:58.000653 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:58.000811 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:58.001168 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:58.001171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:58.001427 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:58.001525 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:58.001531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.002349 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.002354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:58.003008 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:58.003018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.003491 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.003495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:58.003578 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:58.004210 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:58.004223 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:58.006750 (+ 2527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":170,"threads_started":1}
W20260812 11:15:58.007174 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:58.000214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:56237 (local address 127.27.109.62:36075)
0812 11:15:58.000593 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:58.000599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:58.000759 (+ 160us) server_negotiation.cc:407] Connection header received
0812 11:15:58.000982 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:58.000985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:58.001032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:58.001136 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:58.001701 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.002232 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.003137 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.003358 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.003475 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:58.006998 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:56237: BlockingRecv error: recv got EOF from 127.27.109.2:56237 (error 108)
Metrics: {"server-negotiator.queue_time_us":250}
Aug 12 11:15:58 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:58 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:58.268006 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:58.252765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:53603)
0812 11:15:58.253236 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:58.253255 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:58.253450 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:58.253762 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:58.253765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:58.254043 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:58.254147 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:58.254152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.254977 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.254982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:58.255560 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:58.255568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.256082 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.256086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:58.256175 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:58.265218 (+ 9043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:58.265235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:58.267834 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":154,"threads_started":1}
W20260812 11:15:58.268258 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:58.252940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:53603 (local address 127.27.109.62:36075)
0812 11:15:58.253171 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:58.253177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:58.253371 (+ 194us) server_negotiation.cc:407] Connection header received
0812 11:15:58.253576 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:58.253581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:58.253631 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:58.253730 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:58.254277 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.254816 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.255687 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.255912 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.256048 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:58.268047 (+ 11999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:53603: BlockingRecv error: recv got EOF from 127.27.109.3:53603 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 12 11:15:58 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:58 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:58.755107 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:58.746366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:45145)
0812 11:15:58.746868 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:58.746887 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:58.747053 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:58.747564 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:58.747566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:58.747888 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:58.748060 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:58.748066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.749011 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.749015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:58.749745 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:58.749757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.750394 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.750398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:58.750491 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:58.751431 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:58.751445 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:58.754874 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":98,"threads_started":1}
W20260812 11:15:58.755151 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:58.746508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:45145 (local address 127.27.109.62:36075)
0812 11:15:58.746946 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:58.746951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:58.746983 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:15:58.747248 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:58.747251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:58.747314 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:58.747477 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:58.748266 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.748834 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.749926 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:58.750198 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:58.750338 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:58.755034 (+ 4696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:45145: BlockingRecv error: recv got EOF from 127.27.109.1:45145 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 12 11:15:59 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:59 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:59.015465 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.008474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:39301)
0812 11:15:59.008834 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.008860 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.008978 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.009514 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.009517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.009741 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.009862 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.009869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.011103 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.011108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.011695 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.011704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.012210 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.012214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.012301 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.013001 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.013014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.015272 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":148,"threads_started":1}
W20260812 11:15:59.015584 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.008763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39301 (local address 127.27.109.62:36075)
0812 11:15:59.009072 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.009077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.009103 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:15:59.009172 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.009175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.009233 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.009393 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.010076 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.010955 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.011840 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.012073 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.012185 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.015454 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:39301: BlockingRecv error: recv got EOF from 127.27.109.2:39301 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Aug 12 11:15:59 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:59 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:59.277362 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.269159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:38763)
0812 11:15:59.269568 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.269583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.269705 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.270132 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.270135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.270375 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.270614 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.270623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.271674 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.271677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.272363 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.272375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.273030 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.273035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.273117 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.273888 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.273946 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.277189 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
W20260812 11:15:59.277807 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.269353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38763 (local address 127.27.109.62:36075)
0812 11:15:59.269547 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.269552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.269722 (+ 170us) server_negotiation.cc:407] Connection header received
0812 11:15:59.269832 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.269835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.269940 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.270048 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.270800 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.271498 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.272536 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.272813 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.272916 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.277559 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:38763: BlockingRecv error: recv got EOF from 127.27.109.3:38763 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 11:15:59 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:15:59 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:15:59.764093 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.756382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:54109)
0812 11:15:59.756817 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.756841 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.757009 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.757377 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.757383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.757612 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.757716 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.757722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.758533 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.758537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.759214 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.759224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.759723 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.759726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.759818 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.760452 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.760465 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.763882 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":120,"threads_started":1}
W20260812 11:15:59.764670 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.756526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:54109 (local address 127.27.109.62:36075)
0812 11:15:59.756750 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.756756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.756929 (+ 173us) server_negotiation.cc:407] Connection header received
0812 11:15:59.757135 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.757138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.757187 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.757279 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.757847 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.758408 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.759355 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.759581 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.759695 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.764478 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:54109: BlockingRecv error: recv got EOF from 127.27.109.1:54109 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 12 11:16:00 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:16:00 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:16:00.026889 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.016998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:51083)
0812 11:16:00.017326 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:00.017344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:00.017471 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:00.017903 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:00.017906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:00.018159 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:00.018273 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:00.018279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.019556 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.019564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:00.021153 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:00.021191 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.022239 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.022246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:00.022390 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.023474 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:00.023500 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:00.026544 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":127,"threads_started":1}
W20260812 11:16:00.026890 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:00.017437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:51083 (local address 127.27.109.62:36075)
0812 11:16:00.017658 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.017668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.017682 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:16:00.017728 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.017731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.017773 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.017872 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:00.018424 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.019249 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.021451 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.022029 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.022170 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.026731 (+ 4561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:51083: BlockingRecv error: recv got EOF from 127.27.109.2:51083 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 12 11:16:00 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:16:00 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:16:00.287456 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.278645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:44841)
0812 11:16:00.279291 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:00.279315 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:00.279563 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:00.280035 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:00.280039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:00.280540 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:00.280709 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:00.280719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.281925 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.281930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:00.282812 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:00.282852 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.283567 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.283570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:00.283660 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.284455 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:00.284469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:00.287268 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":267,"threads_started":1}
W20260812 11:16:00.287771 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:00.279066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:44841 (local address 127.27.109.62:36075)
0812 11:16:00.279291 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.279299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.279457 (+ 158us) server_negotiation.cc:407] Connection header received
0812 11:16:00.279714 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.279718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.279787 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.279891 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:00.280894 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.281702 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.283095 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.283377 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.283503 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.287589 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:44841: BlockingRecv error: recv got EOF from 127.27.109.3:44841 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 12 11:16:00 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:16:00 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:16:00.774984 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:00.766008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:59553 (local address 127.27.109.62:36075)
0812 11:16:00.766321 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.766325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.766348 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:16:00.766566 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.766569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.766629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.766752 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:00.767493 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.768112 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.769441 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.769762 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.769904 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.774747 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:59553: BlockingRecv error: recv got EOF from 127.27.109.1:59553 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
I20260812 11:16:00.775005 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.765786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:59553)
0812 11:16:00.766209 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:00.766233 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:00.766436 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:00.766964 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:00.766967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:00.767203 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:00.767334 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:00.767340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.768280 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.768284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:00.769251 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:00.769266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.770062 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.770071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:00.770195 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.771105 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:00.771124 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:00.774567 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
Aug 12 11:16:01 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:16:01 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:16:01.037845 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:01.028862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:52697)
0812 11:16:01.029297 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:01.029328 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:01.029528 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:01.030064 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:01.030068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:01.030394 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:01.030543 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:01.030552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.031628 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.031635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:01.032342 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:01.032353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.032928 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.032932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:01.033032 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:01.033895 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:01.033911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:01.037659 (+ 3748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":186,"threads_started":1}
W20260812 11:16:01.038607 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:01.029294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:52697 (local address 127.27.109.62:36075)
0812 11:16:01.029513 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:01.029519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:01.029540 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:16:01.029649 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:01.029652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:01.029715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:01.030075 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:01.030717 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.031479 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.032498 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.032767 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.032898 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:01.038351 (+ 5453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:52697: BlockingRecv error: recv got EOF from 127.27.109.2:52697 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 12 11:16:01 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:16:01 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:16:01.297497 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:01.289035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.3:36463)
0812 11:16:01.289651 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:01.289675 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:01.289880 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:01.290453 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:01.290457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:01.290696 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:01.290953 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:01.290963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.292023 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.292027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:01.292768 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:01.292783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.293523 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.293527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:01.293612 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:01.294398 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:01.294412 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:01.297266 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":259,"threads_started":1}
W20260812 11:16:01.297780 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:01.289208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:36463 (local address 127.27.109.62:36075)
0812 11:16:01.289474 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:01.289481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:01.289854 (+ 373us) server_negotiation.cc:407] Connection header received
0812 11:16:01.290082 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:01.290091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:01.290178 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:01.290306 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:01.291192 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.291844 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.292978 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.293323 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.293466 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:01.297555 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:36463: BlockingRecv error: recv got EOF from 127.27.109.3:36463 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 12 11:16:01 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:16:01 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:16:01.784791 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:01.776382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.1:42063)
0812 11:16:01.776895 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:01.776913 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:01.777055 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:01.777526 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:01.777530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:01.777785 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:01.777929 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:01.777937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.779116 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.779120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:01.779760 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:01.779773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.780430 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.780434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:01.780516 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:01.781630 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:01.781644 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:01.784632 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":240,"threads_started":1}
W20260812 11:16:01.785216 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:01.776606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:42063 (local address 127.27.109.62:36075)
0812 11:16:01.776784 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:01.776788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:01.777042 (+ 254us) server_negotiation.cc:407] Connection header received
0812 11:16:01.777146 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:01.777148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:01.777203 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:01.777300 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:01.778199 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.778904 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.779946 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:01.780216 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:01.780333 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:01.785034 (+ 4701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:42063: BlockingRecv error: recv got EOF from 127.27.109.1:42063 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 11:16:02 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:16:02 dist-test-slave-7z32 krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:16:02.048902 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:02.039819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36075 (local address 127.27.109.2:40255)
0812 11:16:02.040511 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:02.040542 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:02.040731 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:02.041287 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:02.041292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:02.041594 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:02.041731 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:02.041739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.042727 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.042733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:02.043575 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:02.043590 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.044207 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.044212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:02.044317 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:02.045223 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:02.045239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:02.048710 (+ 3471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:36075: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":158,"threads_started":1}
W20260812 11:16:02.049214 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:02.040053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:40255 (local address 127.27.109.62:36075)
0812 11:16:02.040426 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:02.040433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:02.040713 (+ 280us) server_negotiation.cc:407] Connection header received
0812 11:16:02.041003 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:02.041011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:02.041103 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:02.041243 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:02.041895 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.042560 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.043748 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.044032 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.044177 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:02.048942 (+ 4765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:40255: BlockingRecv error: recv got EOF from 127.27.109.2:40255 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
I20260812 11:16:02.126899 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 9226
I20260812 11:16:02.153615 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 9487
I20260812 11:16:02.175010 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 9717
I20260812 11:16:02.193482 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 9105
2026-08-12T11:16:02Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19263 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T11:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:16:02Z Disabled control of system clock
I20260812 11:16:02.249084 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34489
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 11:16:02.391254 10017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:02.391607 10017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:02.391644 10017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:02.396034 10017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:16:02.396126 10017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:02.396142 10017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:02.396160 10017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:16:02.396176 10017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:16:02.401588 10017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:02.402992 10017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:02.404603 10017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:02.423833 10025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:02.423882 10027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:02.423851 10017 server_base.cc:1034] running on GCE node
W20260812 11:16:02.423858 10030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:02.425153 10017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:02.426724 10017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:02.428210 10017 hybrid_clock.cc:648] HybridClock initialized: now 1786533362428175 us; error 83 us; skew 500 ppm
I20260812 11:16:02.431376 10017 webserver.cc:460] Webserver started at http://127.27.109.62:41077/ using document root <none> and password file <none>
I20260812 11:16:02.432355 10017 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:02.432495 10017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:02.432796 10017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:02.435472 10017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "ec4988b7b4f94001bbfda405387def1e"
format_stamp: "Formatted at 2026-08-12 11:16:02 on dist-test-slave-7z32"
server_key: "fa8156ca0fccde6d2c9e4df05ffd72a6"
server_key_iv: "93d65a1833ac980ca1d8cca9980f8eea"
server_key_version: "encryptionkey@0"
I20260812 11:16:02.436709 10017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "ec4988b7b4f94001bbfda405387def1e"
format_stamp: "Formatted at 2026-08-12 11:16:02 on dist-test-slave-7z32"
server_key: "fa8156ca0fccde6d2c9e4df05ffd72a6"
server_key_iv: "93d65a1833ac980ca1d8cca9980f8eea"
server_key_version: "encryptionkey@0"
I20260812 11:16:02.445785 10017 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s
I20260812 11:16:02.450294 10049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:02.452761 10017 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:16:02.453001 10017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "ec4988b7b4f94001bbfda405387def1e"
format_stamp: "Formatted at 2026-08-12 11:16:02 on dist-test-slave-7z32"
server_key: "fa8156ca0fccde6d2c9e4df05ffd72a6"
server_key_iv: "93d65a1833ac980ca1d8cca9980f8eea"
server_key_version: "encryptionkey@0"
I20260812 11:16:02.453189 10017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:02.472940 10017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:02.474174 10017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:02.474428 10017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:02.485924 10017 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:34489
I20260812 11:16:02.486106 10131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:34489 every 8 connection(s)
I20260812 11:16:02.487485 10017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:16:02.492235 10133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:16:02.493017 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 10017
I20260812 11:16:02.493155 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:16:02.493557 28084 external_mini_cluster.cc:1428] Setting key d0ab7ce025e6f44706b467da75d7588c
I20260812 11:16:02.501232 10133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e: Bootstrap starting.
I20260812 11:16:02.504521 10133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e: Neither blocks nor log segments found. Creating new log.
I20260812 11:16:02.505738 10133 log.cc:824] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:16:02.508062 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:02.495638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.27.109.62:34489)
0812 11:16:02.496783 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:02.496810 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:02.496871 (+ 61us) server_negotiation.cc:407] Connection header received
0812 11:16:02.497953 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:02.497984 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:02.498626 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:02.499234 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:02.499894 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.501207 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.502526 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.503304 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.503576 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:02.504971 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:02.504984 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:02.505010 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:02.505651 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:02.505755 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:02.505798 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:02.505980 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:02.506362 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 11:16:02.506777 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":227,"threads_started":1}
I20260812 11:16:02.508777 10133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e: No bootstrap required, opened a new log
I20260812 11:16:02.514330 10133 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4988b7b4f94001bbfda405387def1e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34489 } }
I20260812 11:16:02.514789 10133 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:16:02.514941 10133 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4988b7b4f94001bbfda405387def1e, State: Initialized, Role: FOLLOWER
I20260812 11:16:02.515595 10133 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4988b7b4f94001bbfda405387def1e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34489 } }
I20260812 11:16:02.515776 10133 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:16:02.515873 10133 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:16:02.516017 10133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:16:02.518376 10133 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4988b7b4f94001bbfda405387def1e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34489 } }
I20260812 11:16:02.519470 10133 leader_election.cc:302] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec4988b7b4f94001bbfda405387def1e; no voters:
I20260812 11:16:02.520443 10133 leader_election.cc:288] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:16:02.520629 10140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:16:02.521174 10140 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [term 1 LEADER]: Becoming Leader. State: Replica: ec4988b7b4f94001bbfda405387def1e, State: Running, Role: LEADER
I20260812 11:16:02.521765 10140 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4988b7b4f94001bbfda405387def1e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34489 } }
I20260812 11:16:02.522401 10133 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:16:02.524369 10143 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec4988b7b4f94001bbfda405387def1e. Latest consensus state: current_term: 1 leader_uuid: "ec4988b7b4f94001bbfda405387def1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4988b7b4f94001bbfda405387def1e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34489 } } }
I20260812 11:16:02.524600 10143 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [sys.catalog]: This master's current role is: LEADER
I20260812 11:16:02.525292 10154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:16:02.525820 10142 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec4988b7b4f94001bbfda405387def1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4988b7b4f94001bbfda405387def1e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34489 } } }
I20260812 11:16:02.525981 10142 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e [sys.catalog]: This master's current role is: LEADER
I20260812 11:16:02.529613 10154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:16:02.538170 10154 catalog_manager.cc:1347] Generated new cluster ID: 738a8efbb4444fe98ff60672073846d2
I20260812 11:16:02.538321 10154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:16:02.557893 10154 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:16:02.559508 10154 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:16:02.569823 10154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec4988b7b4f94001bbfda405387def1e: Generated new TSK 0
I20260812 11:16:02.571081 10154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:16:02.587595 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34489
--builtin_ntp_servers=127.27.109.20:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 11:16:02.732086 10167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:02.732492 10167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:02.732537 10167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:02.737309 10167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:02.737416 10167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:02.737476 10167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:16:02.743742 10167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:02.745149 10167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:02.746804 10167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:02.761853 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:02.761572 10176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:02.761477 10177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:02.762414 10167 server_base.cc:1034] running on GCE node
I20260812 11:16:02.763083 10167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:02.764283 10167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:02.767688 10167 hybrid_clock.cc:648] HybridClock initialized: now 1786533362767567 us; error 123 us; skew 500 ppm
I20260812 11:16:02.771003 10167 webserver.cc:460] Webserver started at http://127.27.109.1:35657/ using document root <none> and password file <none>
I20260812 11:16:02.771809 10167 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:02.771908 10167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:02.772179 10167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:02.775199 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "14a97e9dc3a6429081044d1c333ee8d6"
format_stamp: "Formatted at 2026-08-12 11:16:02 on dist-test-slave-7z32"
server_key: "f037b73e7feb63330bf0e4c8d30dc4ae"
server_key_iv: "267059ccda58d318a81386d5262a28a0"
server_key_version: "encryptionkey@0"
I20260812 11:16:02.776073 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "14a97e9dc3a6429081044d1c333ee8d6"
format_stamp: "Formatted at 2026-08-12 11:16:02 on dist-test-slave-7z32"
server_key: "f037b73e7feb63330bf0e4c8d30dc4ae"
server_key_iv: "267059ccda58d318a81386d5262a28a0"
server_key_version: "encryptionkey@0"
I20260812 11:16:02.781075 10167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:16:02.785399 10190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:02.787523 10167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 11:16:02.787734 10167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "14a97e9dc3a6429081044d1c333ee8d6"
format_stamp: "Formatted at 2026-08-12 11:16:02 on dist-test-slave-7z32"
server_key: "f037b73e7feb63330bf0e4c8d30dc4ae"
server_key_iv: "267059ccda58d318a81386d5262a28a0"
server_key_version: "encryptionkey@0"
I20260812 11:16:02.787916 10167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:02.805161 10167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:02.806012 10167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:02.806255 10167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:02.807045 10167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:02.808197 10167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:02.808261 10167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:02.808311 10167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:02.808378 10167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:02.820184 10167 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:38115
I20260812 11:16:02.820312 10371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:38115 every 8 connection(s)
I20260812 11:16:02.821456 10167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:16:02.823045 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 10167
I20260812 11:16:02.823212 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:16:02.823681 28084 external_mini_cluster.cc:1428] Setting key da1d9d1455c1491921dacee2f927ee84
I20260812 11:16:02.826792 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34489
--builtin_ntp_servers=127.27.109.20:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 11:16:02.838363 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:02.824816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33769 (local address 127.27.109.62:34489)
0812 11:16:02.824990 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:02.824997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:02.826794 (+ 1797us) server_negotiation.cc:407] Connection header received
0812 11:16:02.828903 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:02.828908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:02.828991 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:02.829109 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:02.830767 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.831508 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.834793 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.835137 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.835268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:02.837565 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:02.837569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:02.837571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:02.837591 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:02.837621 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:02.837630 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:02.837722 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:02.838108 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 11:16:02.838205 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:16:02.840628 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:02.825323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34489 (local address 127.27.109.1:33769)
0812 11:16:02.826555 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:02.826622 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:02.828558 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:02.829409 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:02.829420 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:02.829551 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:02.830494 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:02.830526 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.831855 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.831863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:02.834509 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:02.834543 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:02.835418 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:02.835426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:02.835794 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:02.836820 (+ 1026us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:16:02.836825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:02.836927 (+ 102us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:16:02.837235 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:02.837779 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:02.837811 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:02.837824 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 11:16:02.838069 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 11:16:02.838703 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":187,"threads_started":1}
I20260812 11:16:02.842625 10372 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34489
I20260812 11:16:02.843330 10372 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:02.844782 10372 heartbeater.cc:511] Master 127.27.109.62:34489 requested a full tablet report, sending...
I20260812 11:16:02.847909 10073 ts_manager.cc:194] Registered new tserver with Master: 14a97e9dc3a6429081044d1c333ee8d6 (127.27.109.1:38115)
I20260812 11:16:02.849293 10073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:33769
W20260812 11:16:02.989575 10380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:02.990058 10380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:02.990139 10380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:02.995405 10380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:02.995560 10380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:02.995678 10380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:16:03.002203 10380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:03.004009 10380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:03.006461 10380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:03.020829 10389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.020902 10392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.021044 10386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:03.021611 10380 server_base.cc:1034] running on GCE node
I20260812 11:16:03.022362 10380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:03.023458 10380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:03.024669 10380 hybrid_clock.cc:648] HybridClock initialized: now 1786533363024643 us; error 82 us; skew 500 ppm
I20260812 11:16:03.027531 10380 webserver.cc:460] Webserver started at http://127.27.109.2:41377/ using document root <none> and password file <none>
I20260812 11:16:03.028414 10380 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:03.028509 10380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:03.028754 10380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:03.031834 10380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "55d0593ed4b142d98895c4d37439b3c5"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "1f8614e3eeb8cd4dc0479fd9b4d775f5"
server_key_iv: "71dff861fef142ef9380a18f978218a9"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.032621 10380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "55d0593ed4b142d98895c4d37439b3c5"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "1f8614e3eeb8cd4dc0479fd9b4d775f5"
server_key_iv: "71dff861fef142ef9380a18f978218a9"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.040048 10380 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 11:16:03.044857 10400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.047488 10380 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.003s
I20260812 11:16:03.047812 10380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "55d0593ed4b142d98895c4d37439b3c5"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "1f8614e3eeb8cd4dc0479fd9b4d775f5"
server_key_iv: "71dff861fef142ef9380a18f978218a9"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.048015 10380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:03.061393 10380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:03.062392 10380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:03.062606 10380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:03.063499 10380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:03.064800 10380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:03.064857 10380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.064905 10380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:03.064921 10380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.079429 10380 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:35703
I20260812 11:16:03.079581 10580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:35703 every 8 connection(s)
I20260812 11:16:03.080814 10380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:16:03.090050 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 10380
I20260812 11:16:03.090188 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:16:03.090623 28084 external_mini_cluster.cc:1428] Setting key 35ac3ec9c492e767ea6db5f39efd5fdf
I20260812 11:16:03.093701 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34489
--builtin_ntp_servers=127.27.109.20:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 11:16:03.094014 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:03.083771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59461 (local address 127.27.109.62:34489)
0812 11:16:03.084027 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:03.084032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:03.085701 (+ 1669us) server_negotiation.cc:407] Connection header received
0812 11:16:03.087608 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:03.087611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:03.087682 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:03.087800 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:03.088911 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.089469 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.091046 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.091389 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.091516 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:03.093384 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:03.093388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:03.093392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:03.093432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:03.093460 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:03.093472 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:03.093572 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:03.093773 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 11:16:03.093859 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 11:16:03.095182 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:03.084390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34489 (local address 127.27.109.2:59461)
0812 11:16:03.085478 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:03.085541 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:03.087217 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:03.088059 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:03.088068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:03.088167 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:03.088736 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:03.088751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.089615 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.089621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:03.090896 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:03.090910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.091721 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.091730 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:03.092033 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:03.093043 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:16:03.093045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:03.093103 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:16:03.093196 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:03.093609 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:03.093618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:03.093623 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:16:03.093775 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 11:16:03.094156 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":187,"threads_started":1}
I20260812 11:16:03.096871 10581 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34489
I20260812 11:16:03.097196 10581 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:03.097903 10581 heartbeater.cc:511] Master 127.27.109.62:34489 requested a full tablet report, sending...
I20260812 11:16:03.099243 10073 ts_manager.cc:194] Registered new tserver with Master: 55d0593ed4b142d98895c4d37439b3c5 (127.27.109.2:35703)
I20260812 11:16:03.099882 10073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:59461
W20260812 11:16:03.246201 10585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:03.246652 10585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:03.246732 10585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:03.253551 10585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:03.253681 10585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:03.253784 10585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:16:03.261332 10585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:03.262741 10585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:03.264436 10585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:03.279132 10592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.279142 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.279146 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:03.279583 10585 server_base.cc:1034] running on GCE node
I20260812 11:16:03.280215 10585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:03.281028 10585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:03.284523 10585 hybrid_clock.cc:648] HybridClock initialized: now 1786533363284435 us; error 100 us; skew 500 ppm
I20260812 11:16:03.292363 10585 webserver.cc:460] Webserver started at http://127.27.109.3:35977/ using document root <none> and password file <none>
I20260812 11:16:03.293174 10585 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:03.293291 10585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:03.293515 10585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:03.296113 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "c48adffd9e7b4b7597823408af5f436e"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "298cd2bab43720bf623dfa8f98895110"
server_key_iv: "47edc9c4fb5af8257c0621483715a3dd"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.296787 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "c48adffd9e7b4b7597823408af5f436e"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "298cd2bab43720bf623dfa8f98895110"
server_key_iv: "47edc9c4fb5af8257c0621483715a3dd"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.301944 10585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:16:03.306468 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.308740 10585 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260812 11:16:03.308995 10585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "c48adffd9e7b4b7597823408af5f436e"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "298cd2bab43720bf623dfa8f98895110"
server_key_iv: "47edc9c4fb5af8257c0621483715a3dd"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.309186 10585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:03.327564 10585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:03.328418 10585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:03.328658 10585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:03.329378 10585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:03.330547 10585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:03.330631 10585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.330708 10585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:03.330753 10585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.344082 10585 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:44769
I20260812 11:16:03.344102 10785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:44769 every 8 connection(s)
I20260812 11:16:03.345343 10585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:16:03.347061 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 10585
I20260812 11:16:03.347231 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:16:03.347628 28084 external_mini_cluster.cc:1428] Setting key 03a6f8909e1d0a954817d0a5b2a37b3a
I20260812 11:16:03.358533 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:03.348487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:49591 (local address 127.27.109.62:34489)
0812 11:16:03.348692 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:03.348698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:03.350094 (+ 1396us) server_negotiation.cc:407] Connection header received
0812 11:16:03.352096 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:03.352100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:03.352186 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:03.352288 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:03.353568 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.354402 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.355977 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.356352 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.356469 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:03.357906 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:03.357910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:03.357913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:03.357942 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:03.357973 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:03.357983 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:03.358082 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:03.358274 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 11:16:03.358367 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:16:03.359592 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:03.348939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34489 (local address 127.27.109.3:49591)
0812 11:16:03.349876 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:03.349938 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:03.351665 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:03.352581 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:03.352589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:03.352697 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:03.353325 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:03.353340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.354568 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.354571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:03.355776 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:03.355789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:03.356573 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:03.356577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:03.356766 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:03.357514 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:16:03.357516 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:03.357561 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:16:03.357642 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:03.358094 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:03.358102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:03.358107 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:16:03.358239 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 11:16:03.358570 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":199,"threads_started":1}
I20260812 11:16:03.361755 10786 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34489
I20260812 11:16:03.362206 10786 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:03.363307 10786 heartbeater.cc:511] Master 127.27.109.62:34489 requested a full tablet report, sending...
I20260812 11:16:03.365306 10073 ts_manager.cc:194] Registered new tserver with Master: c48adffd9e7b4b7597823408af5f436e (127.27.109.3:44769)
I20260812 11:16:03.366050 10073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:49591
I20260812 11:16:03.377300 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:16:03.381628 10138 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:03.380138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.27.109.62:34489)
0812 11:16:03.380398 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:03.380405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:03.380566 (+ 161us) server_negotiation.cc:407] Connection header received
0812 11:16:03.380831 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:03.380835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:03.380906 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:03.381015 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:03.381444 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40216: BlockingRecv error: recv got EOF from 127.0.0.1:40216 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 11:16:03.382364 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 10167
I20260812 11:16:03.418304 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 10380
I20260812 11:16:03.435648 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 10585
I20260812 11:16:03.453450 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 10017
2026-08-12T11:16:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1257 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T11:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:16:03Z Disabled control of system clock
I20260812 11:16:03.505146 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33937
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 11:16:03.636117 10805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:03.636571 10805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:03.636633 10805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:03.641106 10805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:16:03.641211 10805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:03.641227 10805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:03.641250 10805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:16:03.641268 10805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:16:03.646517 10805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:03.647816 10805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:03.649331 10805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:03.662145 10819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.662164 10813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.662145 10817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:03.662760 10805 server_base.cc:1034] running on GCE node
I20260812 11:16:03.663645 10805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:03.665341 10805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:03.666734 10805 hybrid_clock.cc:648] HybridClock initialized: now 1786533363666727 us; error 91 us; skew 500 ppm
I20260812 11:16:03.669889 10805 webserver.cc:460] Webserver started at http://127.27.109.62:39053/ using document root <none> and password file <none>
I20260812 11:16:03.671010 10805 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:03.671152 10805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:03.671476 10805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:03.674686 10805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "6a2db750ad104373bc6da4568fc1f3d5"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "dbda2fb2c461d63928842b48cdbec3a4"
server_key_iv: "dd25ce67a9b326a38696831f1224a45d"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.675655 10805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "6a2db750ad104373bc6da4568fc1f3d5"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "dbda2fb2c461d63928842b48cdbec3a4"
server_key_iv: "dd25ce67a9b326a38696831f1224a45d"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.682653 10805 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 11:16:03.688210 10827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.690678 10805 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:16:03.691068 10805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "6a2db750ad104373bc6da4568fc1f3d5"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "dbda2fb2c461d63928842b48cdbec3a4"
server_key_iv: "dd25ce67a9b326a38696831f1224a45d"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.691309 10805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:03.698966 10805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:03.700044 10805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:03.700321 10805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:03.711654 10805 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:33937
I20260812 11:16:03.711648 10906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:33937 every 8 connection(s)
I20260812 11:16:03.713078 10805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:16:03.715399 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 10805
I20260812 11:16:03.715517 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:16:03.716032 28084 external_mini_cluster.cc:1428] Setting key f1f00598ee4bfc1302ae0162e794e98e
I20260812 11:16:03.717924 10907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:16:03.724793 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:03.718247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.27.109.62:33937)
0812 11:16:03.719385 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:03.719411 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:03.719469 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:16:03.720442 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:03.720466 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:03.720984 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:03.721408 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:03.722739 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:03.722746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:03.722765 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:03.722781 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:03.722942 (+ 161us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:03.722967 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:03.723150 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:03.723278 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 11:16:03.723764 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":190,"threads_started":1}
I20260812 11:16:03.726446 10907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5: Bootstrap starting.
I20260812 11:16:03.729238 10907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5: Neither blocks nor log segments found. Creating new log.
I20260812 11:16:03.730149 10907 log.cc:824] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:16:03.733623 10907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5: No bootstrap required, opened a new log
I20260812 11:16:03.738093 10907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2db750ad104373bc6da4568fc1f3d5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33937 } }
I20260812 11:16:03.738940 10907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:16:03.739091 10907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2db750ad104373bc6da4568fc1f3d5, State: Initialized, Role: FOLLOWER
I20260812 11:16:03.739676 10907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2db750ad104373bc6da4568fc1f3d5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33937 } }
I20260812 11:16:03.739880 10907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:16:03.739979 10907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:16:03.740115 10907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:16:03.741415 10907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2db750ad104373bc6da4568fc1f3d5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33937 } }
I20260812 11:16:03.741868 10907 leader_election.cc:302] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a2db750ad104373bc6da4568fc1f3d5; no voters:
I20260812 11:16:03.742321 10907 leader_election.cc:288] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:16:03.742540 10913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:16:03.743022 10913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 6a2db750ad104373bc6da4568fc1f3d5, State: Running, Role: LEADER
I20260812 11:16:03.743546 10913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2db750ad104373bc6da4568fc1f3d5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33937 } }
I20260812 11:16:03.743979 10907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:16:03.746451 10914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a2db750ad104373bc6da4568fc1f3d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2db750ad104373bc6da4568fc1f3d5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33937 } } }
I20260812 11:16:03.746711 10914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:16:03.746451 10916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a2db750ad104373bc6da4568fc1f3d5. Latest consensus state: current_term: 1 leader_uuid: "6a2db750ad104373bc6da4568fc1f3d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2db750ad104373bc6da4568fc1f3d5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33937 } } }
I20260812 11:16:03.746980 10916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:16:03.747749 10933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:16:03.752086 10933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:16:03.760862 10933 catalog_manager.cc:1347] Generated new cluster ID: e9d7452bc56f4ca6a2f33bb9c94b902a
I20260812 11:16:03.760980 10933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:16:03.773695 10933 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:16:03.774379 10933 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:16:03.782508 10933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a2db750ad104373bc6da4568fc1f3d5: Generated new TSK 0
I20260812 11:16:03.783533 10933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:16:03.796150 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33937
--builtin_ntp_servers=127.27.109.20:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 11:16:03.938299 10941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:03.938673 10941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:03.938717 10941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:03.944121 10941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:03.944597 10941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:03.945539 10941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:16:03.952500 10941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:03.953981 10941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:03.956112 10941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:03.969835 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.969944 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:03.969905 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:03.970297 10941 server_base.cc:1034] running on GCE node
I20260812 11:16:03.971365 10941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:03.972407 10941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:03.975924 10941 hybrid_clock.cc:648] HybridClock initialized: now 1786533363975849 us; error 87 us; skew 500 ppm
I20260812 11:16:03.978725 10941 webserver.cc:460] Webserver started at http://127.27.109.1:38259/ using document root <none> and password file <none>
I20260812 11:16:03.979537 10941 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:03.979655 10941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:03.979925 10941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:03.982337 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "1a645f67e826480c91b36b1072fae751"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "2417db5ba452ef29d8c7619dd8151e41"
server_key_iv: "b26ce72b25d4bf7bbc4753423d1165ed"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.983083 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "1a645f67e826480c91b36b1072fae751"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "2417db5ba452ef29d8c7619dd8151e41"
server_key_iv: "b26ce72b25d4bf7bbc4753423d1165ed"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.988559 10941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:16:03.993152 10959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:03.995498 10941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:16:03.995756 10941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "1a645f67e826480c91b36b1072fae751"
format_stamp: "Formatted at 2026-08-12 11:16:03 on dist-test-slave-7z32"
server_key: "2417db5ba452ef29d8c7619dd8151e41"
server_key_iv: "b26ce72b25d4bf7bbc4753423d1165ed"
server_key_version: "encryptionkey@0"
I20260812 11:16:03.995966 10941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:04.002627 10941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:04.004014 10941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:04.004359 10941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:04.005479 10941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:04.007467 10941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:04.007576 10941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.007660 10941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:04.007705 10941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.025103 10941 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:35851
I20260812 11:16:04.025163 11143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:35851 every 8 connection(s)
I20260812 11:16:04.026526 10941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:16:04.027823 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 10941
I20260812 11:16:04.027962 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:16:04.028357 28084 external_mini_cluster.cc:1428] Setting key 0e3df1718e78c503f2ed4bb7f23f346b
I20260812 11:16:04.032197 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33937
--builtin_ntp_servers=127.27.109.20:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 11:16:04.039748 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:04.031097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:34605 (local address 127.27.109.62:33937)
0812 11:16:04.031332 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:04.031337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:04.033800 (+ 2463us) server_negotiation.cc:407] Connection header received
0812 11:16:04.036706 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:04.036711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:04.036793 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:04.036909 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:04.038883 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:04.038891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:04.038899 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:04.038928 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:04.038970 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:04.038986 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:04.039145 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:04.039392 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 11:16:04.039475 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260812 11:16:04.040795 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:04.031595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33937 (local address 127.27.109.1:34605)
0812 11:16:04.033067 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:04.033511 (+ 444us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:04.036235 (+ 2724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:04.037331 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:04.037344 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:04.037502 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:04.038414 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:16:04.038415 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:04.038470 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:16:04.038570 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:04.039134 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:04.039144 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:04.039149 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:16:04.039482 (+ 333us) client_negotiation.cc:240] Negotiation successful
0812 11:16:04.039759 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":325,"threads_started":1}
I20260812 11:16:04.042850 11148 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33937
I20260812 11:16:04.043187 11148 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:04.043766 11148 heartbeater.cc:511] Master 127.27.109.62:33937 requested a full tablet report, sending...
I20260812 11:16:04.046445 10852 ts_manager.cc:194] Registered new tserver with Master: 1a645f67e826480c91b36b1072fae751 (127.27.109.1:35851)
W20260812 11:16:04.196815 11151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:04.197381 11151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:04.197467 11151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:04.203738 11151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:04.203905 11151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:04.204011 11151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:16:04.210496 11151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:04.212352 11151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:04.214355 11151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:04.230072 11164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:04.230108 11161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:04.230072 11162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:04.230675 11151 server_base.cc:1034] running on GCE node
I20260812 11:16:04.231508 11151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:04.232636 11151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:04.234081 11151 hybrid_clock.cc:648] HybridClock initialized: now 1786533364233963 us; error 128 us; skew 500 ppm
I20260812 11:16:04.237578 11151 webserver.cc:460] Webserver started at http://127.27.109.2:36817/ using document root <none> and password file <none>
I20260812 11:16:04.238497 11151 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:04.238597 11151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:04.238957 11151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:04.242630 11151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "35e1a78fe45b4a77bb41a3597ec9fb88"
format_stamp: "Formatted at 2026-08-12 11:16:04 on dist-test-slave-7z32"
server_key: "dff2d59a44644e93b5c29da578351ae3"
server_key_iv: "4a793ef4d912801f30bd3a7fc6330b2c"
server_key_version: "encryptionkey@0"
I20260812 11:16:04.244408 11151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "35e1a78fe45b4a77bb41a3597ec9fb88"
format_stamp: "Formatted at 2026-08-12 11:16:04 on dist-test-slave-7z32"
server_key: "dff2d59a44644e93b5c29da578351ae3"
server_key_iv: "4a793ef4d912801f30bd3a7fc6330b2c"
server_key_version: "encryptionkey@0"
I20260812 11:16:04.252772 11151 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260812 11:16:04.258173 11172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.260591 11151 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:16:04.260818 11151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "35e1a78fe45b4a77bb41a3597ec9fb88"
format_stamp: "Formatted at 2026-08-12 11:16:04 on dist-test-slave-7z32"
server_key: "dff2d59a44644e93b5c29da578351ae3"
server_key_iv: "4a793ef4d912801f30bd3a7fc6330b2c"
server_key_version: "encryptionkey@0"
I20260812 11:16:04.261000 11151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:04.267812 11151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:04.268740 11151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:04.268949 11151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:04.269670 11151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:04.271312 11151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:04.271394 11151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.271446 11151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:04.271463 11151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.285975 11151 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:41841
I20260812 11:16:04.286085 11352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:41841 every 8 connection(s)
I20260812 11:16:04.287421 11151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:16:04.295601 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 11151
I20260812 11:16:04.295745 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:16:04.296123 28084 external_mini_cluster.cc:1428] Setting key f5d8ffb06e4e64b99fe8b78f521f30c9
I20260812 11:16:04.297590 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:04.290307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39439 (local address 127.27.109.62:33937)
0812 11:16:04.290697 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:04.290702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:04.292548 (+ 1846us) server_negotiation.cc:407] Connection header received
0812 11:16:04.294169 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:04.294175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:04.294250 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:04.294392 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:04.296877 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:04.296881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:04.296884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:04.296904 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:04.296945 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:04.296957 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:04.297071 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:04.297359 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 11:16:04.297430 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260812 11:16:04.299099 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33937
--builtin_ntp_servers=127.27.109.20:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 11:16:04.299588 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:04.291063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33937 (local address 127.27.109.2:39439)
0812 11:16:04.292167 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:04.292233 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:04.293816 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:04.294822 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:04.294860 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:04.295030 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:04.296322 (+ 1292us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:16:04.296325 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:04.296406 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:16:04.296601 (+ 195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:04.297093 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:04.297107 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:04.297117 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:16:04.297402 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 11:16:04.298040 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":202,"threads_started":1}
I20260812 11:16:04.301486 11353 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33937
I20260812 11:16:04.301920 11353 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:04.302561 11353 heartbeater.cc:511] Master 127.27.109.62:33937 requested a full tablet report, sending...
I20260812 11:16:04.304735 10852 ts_manager.cc:194] Registered new tserver with Master: 35e1a78fe45b4a77bb41a3597ec9fb88 (127.27.109.2:41841)
W20260812 11:16:04.453649 11356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:04.454005 11356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:04.454049 11356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:04.460220 11356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:04.460350 11356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:04.460425 11356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:16:04.466921 11356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:04.468394 11356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:04.470072 11356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:04.484016 11366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:04.484036 11370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:04.484243 11356 server_base.cc:1034] running on GCE node
W20260812 11:16:04.484018 11367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:04.485126 11356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:04.486007 11356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:04.487541 11356 hybrid_clock.cc:648] HybridClock initialized: now 1786533364487507 us; error 57 us; skew 500 ppm
I20260812 11:16:04.490564 11356 webserver.cc:460] Webserver started at http://127.27.109.3:37745/ using document root <none> and password file <none>
I20260812 11:16:04.491400 11356 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:04.491513 11356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:04.491796 11356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:04.494246 11356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "d5ed3663bab64340a9a6ec9262c57b55"
format_stamp: "Formatted at 2026-08-12 11:16:04 on dist-test-slave-7z32"
server_key: "df4cf319f476702c6ed97e26a1a33576"
server_key_iv: "88c4d62a95676314c7828bb9eda7f2ac"
server_key_version: "encryptionkey@0"
I20260812 11:16:04.495078 11356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "d5ed3663bab64340a9a6ec9262c57b55"
format_stamp: "Formatted at 2026-08-12 11:16:04 on dist-test-slave-7z32"
server_key: "df4cf319f476702c6ed97e26a1a33576"
server_key_iv: "88c4d62a95676314c7828bb9eda7f2ac"
server_key_version: "encryptionkey@0"
I20260812 11:16:04.500630 11356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:16:04.504925 11379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.507635 11356 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:16:04.507885 11356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "d5ed3663bab64340a9a6ec9262c57b55"
format_stamp: "Formatted at 2026-08-12 11:16:04 on dist-test-slave-7z32"
server_key: "df4cf319f476702c6ed97e26a1a33576"
server_key_iv: "88c4d62a95676314c7828bb9eda7f2ac"
server_key_version: "encryptionkey@0"
I20260812 11:16:04.508078 11356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:04.517045 11356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:04.517951 11356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:04.518213 11356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:04.519038 11356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:04.520387 11356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:04.520470 11356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.520546 11356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:04.520604 11356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:04.534436 11356 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:37737
I20260812 11:16:04.534519 11567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:37737 every 8 connection(s)
I20260812 11:16:04.535743 11356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:16:04.541711 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 11356
I20260812 11:16:04.541893 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:16:04.542229 28084 external_mini_cluster.cc:1428] Setting key f566d933de5c5a0644f3540c8b891f5c
I20260812 11:16:04.546468 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:04.538766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:53629 (local address 127.27.109.62:33937)
0812 11:16:04.539048 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:04.539054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:04.540957 (+ 1903us) server_negotiation.cc:407] Connection header received
0812 11:16:04.543163 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:04.543168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:04.543234 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:04.543389 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:04.545685 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:04.545689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:04.545691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:16:04.545707 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:04.545737 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:16:04.545747 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:04.545861 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:04.546227 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 11:16:04.546305 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260812 11:16:04.548514 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:04.539315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33937 (local address 127.27.109.3:53629)
0812 11:16:04.540578 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:04.540654 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:04.542591 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:04.543854 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:04.543899 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:04.544131 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:04.545116 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:16:04.545118 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:04.545257 (+ 139us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:16:04.545420 (+ 163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:04.545896 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:04.545908 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:04.545917 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:16:04.546236 (+ 319us) client_negotiation.cc:240] Negotiation successful
0812 11:16:04.546674 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":174,"threads_started":1}
I20260812 11:16:04.550792 11571 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33937
I20260812 11:16:04.551263 11571 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:04.551988 11571 heartbeater.cc:511] Master 127.27.109.62:33937 requested a full tablet report, sending...
I20260812 11:16:04.554554 10852 ts_manager.cc:194] Registered new tserver with Master: d5ed3663bab64340a9a6ec9262c57b55 (127.27.109.3:37737)
I20260812 11:16:04.561450 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:16:04.566282 10911 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:04.564632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.27.109.62:33937)
0812 11:16:04.564910 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:04.564918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:04.565138 (+ 220us) server_negotiation.cc:407] Connection header received
0812 11:16:04.565375 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:04.565378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:04.565506 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:04.565633 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:04.566107 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49644: BlockingRecv error: recv got EOF from 127.0.0.1:49644 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 11:16:04.567174 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 10941
I20260812 11:16:04.584437 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 11151
I20260812 11:16:04.606001 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 11356
I20260812 11:16:04.632899 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 10805
2026-08-12T11:16:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1187 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:16:04 dist-test-slave-7z32 krb5kdc[11584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:16:04 dist-test-slave-7z32 krb5kdc[11584](info): set up 2 sockets
Aug 12 11:16:04 dist-test-slave-7z32 krb5kdc[11584](info): commencing operation
krb5kdc: starting...
W20260812 11:16:06.163522 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.485s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:16:06 dist-test-slave-7z32 krb5kdc[11584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:16:06.362314 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36017
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:16:06.523948 11598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:06.524411 11598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:06.524472 11598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:06.529757 11598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:16:06.529913 11598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:06.529937 11598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:06.529958 11598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:16:06.529974 11598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:16:06.541528 11598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:06.543383 11598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:06.545148 11598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:06.560132 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:06.560161 11612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:06.560161 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:06.561913 11598 server_base.cc:1034] running on GCE node
I20260812 11:16:06.563218 11598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:06.565464 11598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:06.568974 11598 hybrid_clock.cc:648] HybridClock initialized: now 1786533366568882 us; error 80 us; skew 500 ppm
Aug 12 11:16:06 dist-test-slave-7z32 krb5kdc[11584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:16:06.573879 11598 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:16:06.575754 11598 webserver.cc:460] Webserver started at http://127.27.109.62:37791/ using document root <none> and password file <none>
I20260812 11:16:06.576611 11598 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:06.576707 11598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:06.576954 11598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:06.580116 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "7de7421c15a04f65a0aeac2b161056b8"
format_stamp: "Formatted at 2026-08-12 11:16:06 on dist-test-slave-7z32"
server_key: "163a9f7039de70cacd6afff3dd01f177"
server_key_iv: "630cd92f5697ece3ead7449c1f6ac3a8"
server_key_version: "encryptionkey@0"
I20260812 11:16:06.580938 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "7de7421c15a04f65a0aeac2b161056b8"
format_stamp: "Formatted at 2026-08-12 11:16:06 on dist-test-slave-7z32"
server_key: "163a9f7039de70cacd6afff3dd01f177"
server_key_iv: "630cd92f5697ece3ead7449c1f6ac3a8"
server_key_version: "encryptionkey@0"
I20260812 11:16:06.587209 11598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:16:06.591949 11625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:06.594322 11598 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:16:06.594544 11598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "7de7421c15a04f65a0aeac2b161056b8"
format_stamp: "Formatted at 2026-08-12 11:16:06 on dist-test-slave-7z32"
server_key: "163a9f7039de70cacd6afff3dd01f177"
server_key_iv: "630cd92f5697ece3ead7449c1f6ac3a8"
server_key_version: "encryptionkey@0"
I20260812 11:16:06.594717 11598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:06.628866 11598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:06.635043 11598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:06.635321 11598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:06.646739 11598 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:36017
I20260812 11:16:06.647171 11717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:36017 every 8 connection(s)
I20260812 11:16:06.648562 11598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:16:06.652483 11718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:16:06.656875 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 11598
I20260812 11:16:06.657027 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:16:06.657330 28084 external_mini_cluster.cc:1428] Setting key 3c10b55a13f45ae0e740d5d9f72bdb5d
I20260812 11:16:06.659858 11718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8: Bootstrap starting.
I20260812 11:16:06.662673 11718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8: Neither blocks nor log segments found. Creating new log.
I20260812 11:16:06.663664 11718 log.cc:824] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:16:06.665987 11718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8: No bootstrap required, opened a new log
Aug 12 11:16:06 dist-test-slave-7z32 krb5kdc[11584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:16:06.672076 11718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7421c15a04f65a0aeac2b161056b8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36017 } }
I20260812 11:16:06.672446 11718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:16:06.672523 11718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de7421c15a04f65a0aeac2b161056b8, State: Initialized, Role: FOLLOWER
I20260812 11:16:06.673164 11718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7421c15a04f65a0aeac2b161056b8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36017 } }
I20260812 11:16:06.673426 11718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:16:06.673516 11718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:16:06.673660 11718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:16:06.675221 11718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7421c15a04f65a0aeac2b161056b8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36017 } }
I20260812 11:16:06.675750 11718 leader_election.cc:302] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de7421c15a04f65a0aeac2b161056b8; no voters:
I20260812 11:16:06.676240 11718 leader_election.cc:288] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:16:06.676506 11726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:16:06.676894 11726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7de7421c15a04f65a0aeac2b161056b8, State: Running, Role: LEADER
I20260812 11:16:06.677481 11726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7421c15a04f65a0aeac2b161056b8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36017 } }
I20260812 11:16:06.678599 11718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:16:06.680287 11728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de7421c15a04f65a0aeac2b161056b8. Latest consensus state: current_term: 1 leader_uuid: "7de7421c15a04f65a0aeac2b161056b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7421c15a04f65a0aeac2b161056b8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36017 } } }
I20260812 11:16:06.680546 11728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:16:06.680295 11727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de7421c15a04f65a0aeac2b161056b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7421c15a04f65a0aeac2b161056b8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36017 } } }
I20260812 11:16:06.681011 11727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:16:06.680962 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:06.659824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.27.109.62:36017)
0812 11:16:06.660599 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:06.660635 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:06.660699 (+ 64us) server_negotiation.cc:407] Connection header received
0812 11:16:06.661756 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:06.661786 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:06.662336 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:06.662837 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:06.664051 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:06.665263 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:06.666370 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:06.667225 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:06.667463 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:06.671703 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:06.671712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:06.671741 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:16:06.671797 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:06.674432 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:06.675382 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:06.675390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:06.675399 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:06.675600 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:06.675988 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:06.675992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:06.675993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:06.676834 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:06.678396 (+ 1562us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:06.678630 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 11:16:06.679217 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":224,"threads_started":1}
I20260812 11:16:06.681576 11736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:16:06.686605 11736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:16:06.695616 11736 catalog_manager.cc:1347] Generated new cluster ID: 1dbddcaba9e2406e86e67a0cbe09a41d
I20260812 11:16:06.695757 11736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:16:06.737135 11736 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:16:06.738427 11736 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:16:06.747613 11736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8: Generated new TSK 0
I20260812 11:16:06.748653 11736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:16:06.796847 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36017
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:16:06.944352 11753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:06.944706 11753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:06.944754 11753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:06.949291 11753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:06.949385 11753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:06.949451 11753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:16:06.955435 11753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:06.957142 11753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:06.958437 11753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:06.972813 11763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:06.972859 11764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:06.973043 11753 server_base.cc:1034] running on GCE node
W20260812 11:16:06.973487 11770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:06.974079 11753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:06.975063 11753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:06.978539 11753 hybrid_clock.cc:648] HybridClock initialized: now 1786533366978431 us; error 104 us; skew 500 ppm
Aug 12 11:16:06 dist-test-slave-7z32 krb5kdc[11584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:16:06.983639 11753 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:16:06.985674 11753 webserver.cc:460] Webserver started at http://127.27.109.1:37655/ using document root <none> and password file <none>
I20260812 11:16:06.986774 11753 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:06.986953 11753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:06.987458 11753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:06.992743 11753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "b51027dcac654d8d8de2a694292fe4ba"
format_stamp: "Formatted at 2026-08-12 11:16:06 on dist-test-slave-7z32"
server_key: "dfb7ae9c9f1d01419a5c4cfc70892562"
server_key_iv: "083b9ac48ff4e351206bd331ec2b08d8"
server_key_version: "encryptionkey@0"
I20260812 11:16:06.993777 11753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "b51027dcac654d8d8de2a694292fe4ba"
format_stamp: "Formatted at 2026-08-12 11:16:06 on dist-test-slave-7z32"
server_key: "dfb7ae9c9f1d01419a5c4cfc70892562"
server_key_iv: "083b9ac48ff4e351206bd331ec2b08d8"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.001402 11753 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 11:16:07.006265 11780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.008945 11753 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:16:07.009281 11753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "b51027dcac654d8d8de2a694292fe4ba"
format_stamp: "Formatted at 2026-08-12 11:16:06 on dist-test-slave-7z32"
server_key: "dfb7ae9c9f1d01419a5c4cfc70892562"
server_key_iv: "083b9ac48ff4e351206bd331ec2b08d8"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.009511 11753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:07.026017 11753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:07.033000 11753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:07.033344 11753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:07.034282 11753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:07.035701 11753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:07.035795 11753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.035890 11753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:07.035934 11753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.051437 11753 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:44631
I20260812 11:16:07.051568 11961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:44631 every 8 connection(s)
I20260812 11:16:07.052856 11753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:16:07.058431 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 11753
I20260812 11:16:07.058743 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:16:07.059211 28084 external_mini_cluster.cc:1428] Setting key f59d84b6b5372b6bb07666d65aa30f48
Aug 12 11:16:07 dist-test-slave-7z32 krb5kdc[11584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:16:07.071995 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.056000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:56547 (local address 127.27.109.62:36017)
0812 11:16:07.056233 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:07.056239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:07.057608 (+ 1369us) server_negotiation.cc:407] Connection header received
0812 11:16:07.059519 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:07.059523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:07.059595 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:07.059709 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:07.061624 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.062217 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.063602 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.063852 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.063976 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.067185 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:07.067188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:07.067191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:16:07.067266 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:07.069372 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.070064 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.070067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.070069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.070157 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.070563 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.070567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.070569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.070794 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:07.070976 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:07.071612 (+ 636us) server_negotiation.cc:299] Negotiation successful
0812 11:16:07.071781 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 11:16:07.072939 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.056589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36017 (local address 127.27.109.1:56547)
0812 11:16:07.057360 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:07.057446 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:07.059211 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:07.059938 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:07.059970 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:07.060520 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:07.061434 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.061455 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.062377 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.062381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.063450 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.063462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.064037 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.064041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.064228 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.064942 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:07.064960 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:07.066954 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:07.069546 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:16:07.069555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:16:07.069583 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:16:07.069916 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:16:07.070262 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:16:07.070265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:16:07.070267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:16:07.070447 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:16:07.070987 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:07.070997 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:07.071330 (+ 333us) client_negotiation.cc:769] Sending connection context
0812 11:16:07.071549 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 11:16:07.071958 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":209,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:16:07.075254 11962 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36017
I20260812 11:16:07.075755 11962 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:07.076473 11962 heartbeater.cc:511] Master 127.27.109.62:36017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:16:07.079514 11653 ts_manager.cc:194] Registered new tserver with Master: b51027dcac654d8d8de2a694292fe4ba (127.27.109.1:44631)
I20260812 11:16:07.081758 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:56547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:16:07.104163 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36017
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:16:07.258652 11971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:07.259119 11971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:07.259207 11971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:07.263546 11971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:07.263657 11971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:07.263761 11971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:16:07.269692 11971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:07.271287 11971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:07.272739 11971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:07.287742 11979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:07.287781 11978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:07.287810 11971 server_base.cc:1034] running on GCE node
W20260812 11:16:07.287731 11982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:07.289060 11971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:07.290091 11971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:07.293494 11971 hybrid_clock.cc:648] HybridClock initialized: now 1786533367293396 us; error 86 us; skew 500 ppm
Aug 12 11:16:07 dist-test-slave-7z32 krb5kdc[11584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:16:07.298341 11971 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:16:07.299916 11971 webserver.cc:460] Webserver started at http://127.27.109.2:38441/ using document root <none> and password file <none>
I20260812 11:16:07.301003 11971 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:07.301260 11971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:07.301718 11971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:07.305352 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "830401feab7245d2a491acdac788c4d0"
format_stamp: "Formatted at 2026-08-12 11:16:07 on dist-test-slave-7z32"
server_key: "d7ed7b9a1c91177a269159802d3c6aae"
server_key_iv: "2ac0c1a592e9aab050d5eb2593c88e2e"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.307189 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "830401feab7245d2a491acdac788c4d0"
format_stamp: "Formatted at 2026-08-12 11:16:07 on dist-test-slave-7z32"
server_key: "d7ed7b9a1c91177a269159802d3c6aae"
server_key_iv: "2ac0c1a592e9aab050d5eb2593c88e2e"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.314714 11971 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 11:16:07.319062 11993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.321949 11971 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260812 11:16:07.322268 11971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "830401feab7245d2a491acdac788c4d0"
format_stamp: "Formatted at 2026-08-12 11:16:07 on dist-test-slave-7z32"
server_key: "d7ed7b9a1c91177a269159802d3c6aae"
server_key_iv: "2ac0c1a592e9aab050d5eb2593c88e2e"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.322494 11971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:07.352634 11971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:07.358132 11971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:07.358491 11971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:07.359480 11971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:07.360774 11971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:07.360865 11971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.360944 11971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:07.360991 11971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.378347 11971 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44041
I20260812 11:16:07.378400 12191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44041 every 8 connection(s)
I20260812 11:16:07.380270 11971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:16:07.388770 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 11971
I20260812 11:16:07.388968 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:16:07.389350 28084 external_mini_cluster.cc:1428] Setting key fdc751b036bb3d500cbb73aa07164084
Aug 12 11:16:07 dist-test-slave-7z32 krb5kdc[11584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:16:07.404270 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.383398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39715 (local address 127.27.109.62:36017)
0812 11:16:07.383660 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:07.383665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:07.385075 (+ 1410us) server_negotiation.cc:407] Connection header received
0812 11:16:07.387217 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:07.387221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:07.387308 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:07.387432 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:07.389227 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.390112 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.392296 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.392566 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.392699 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.397226 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:07.397236 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:07.397243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:16:07.397306 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:07.400346 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.401534 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.401542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.401547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.401730 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.402305 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.402311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.402316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.402710 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:07.402888 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:07.403796 (+ 908us) server_negotiation.cc:299] Negotiation successful
0812 11:16:07.404030 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260812 11:16:07.405416 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.383786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36017 (local address 127.27.109.2:39715)
0812 11:16:07.384756 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:07.384876 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:07.386884 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:07.387694 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:07.387704 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:07.388238 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:07.388990 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.389016 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.390422 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.390433 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.392081 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.392103 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.392794 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.392800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.393035 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.394003 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:07.394025 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:07.396779 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:07.400649 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:16:07.400667 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:16:07.400689 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:16:07.401248 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:16:07.401893 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:16:07.401896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:16:07.401898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:16:07.402150 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:16:07.402992 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:07.403000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:07.403412 (+ 412us) client_negotiation.cc:769] Sending connection context
0812 11:16:07.403642 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 11:16:07.404232 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":212,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:16:07.407433 12194 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36017
I20260812 11:16:07.407871 12194 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:07.408926 12194 heartbeater.cc:511] Master 127.27.109.62:36017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:16:07.410866 11653 ts_manager.cc:194] Registered new tserver with Master: 830401feab7245d2a491acdac788c4d0 (127.27.109.2:44041)
I20260812 11:16:07.411917 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:39715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:16:07.437317 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36017
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:16:07.578379 12200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:16:07.578811 12200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:16:07.578938 12200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:16:07.583521 12200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:16:07.583614 12200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:16:07.583679 12200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:16:07.589443 12200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:16:07.590729 12200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:16:07.592458 12200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:16:07.606076 12215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:07.606235 12213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:16:07.606076 12212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:16:07.607204 12200 server_base.cc:1034] running on GCE node
I20260812 11:16:07.608002 12200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:16:07.609001 12200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:16:07.612380 12200 hybrid_clock.cc:648] HybridClock initialized: now 1786533367612298 us; error 95 us; skew 500 ppm
Aug 12 11:16:07 dist-test-slave-7z32 krb5kdc[11584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:16:07.616410 12200 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:16:07.617703 12200 webserver.cc:460] Webserver started at http://127.27.109.3:35669/ using document root <none> and password file <none>
I20260812 11:16:07.618484 12200 fs_manager.cc:359] Metadata directory not provided
I20260812 11:16:07.618793 12200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:16:07.619212 12200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:16:07.621861 12200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "eea454f3e8f140499d159877ca5165cd"
format_stamp: "Formatted at 2026-08-12 11:16:07 on dist-test-slave-7z32"
server_key: "ef1de9f443ef07227c96616b6166a83c"
server_key_iv: "11dda454ff83b1141087dc782e431509"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.622776 12200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "eea454f3e8f140499d159877ca5165cd"
format_stamp: "Formatted at 2026-08-12 11:16:07 on dist-test-slave-7z32"
server_key: "ef1de9f443ef07227c96616b6166a83c"
server_key_iv: "11dda454ff83b1141087dc782e431509"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.628535 12200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:16:07.632477 12223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.634433 12200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:16:07.634609 12200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "eea454f3e8f140499d159877ca5165cd"
format_stamp: "Formatted at 2026-08-12 11:16:07 on dist-test-slave-7z32"
server_key: "ef1de9f443ef07227c96616b6166a83c"
server_key_iv: "11dda454ff83b1141087dc782e431509"
server_key_version: "encryptionkey@0"
I20260812 11:16:07.634902 12200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:16:07.650439 12200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:16:07.654793 12200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:16:07.655139 12200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:16:07.655980 12200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:16:07.657080 12200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:16:07.657150 12200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.657232 12200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:16:07.657276 12200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:16:07.670642 12200 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:37805
I20260812 11:16:07.670876 12401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:37805 every 8 connection(s)
I20260812 11:16:07.672087 12200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:16:07.680938 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 12200
I20260812 11:16:07.681152 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:16:07.681559 28084 external_mini_cluster.cc:1428] Setting key c537c3de69c52d0856bc4b414b4c8216
Aug 12 11:16:07 dist-test-slave-7z32 krb5kdc[11584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:16:07.695101 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.677692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:60473 (local address 127.27.109.62:36017)
0812 11:16:07.677926 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:07.677931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:07.677966 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:16:07.679862 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:07.679867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:07.679956 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:07.680195 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:07.682353 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.683427 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.685348 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.685682 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.685878 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.689773 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:07.689778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:07.689781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:16:07.689836 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:07.691703 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.692727 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.692731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.692732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.692837 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.693391 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.693394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.693395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.693664 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:07.693765 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:07.694661 (+ 896us) server_negotiation.cc:299] Negotiation successful
0812 11:16:07.694925 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:16:07.696871 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.676244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36017 (local address 127.27.109.3:60473)
0812 11:16:07.677495 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:07.677570 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:07.679457 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:07.680612 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:07.680622 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:07.681202 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:16:07.682065 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.682091 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.683766 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.683771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.685109 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.685130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.685978 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.685982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.686277 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.687211 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:16:07.687235 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:16:07.689453 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:16:07.691972 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:16:07.691982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:16:07.692010 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:16:07.692488 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:16:07.693004 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:16:07.693009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:16:07.693011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:16:07.693259 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:16:07.693837 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:16:07.693845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:16:07.694301 (+ 456us) client_negotiation.cc:769] Sending connection context
0812 11:16:07.694577 (+ 276us) client_negotiation.cc:240] Negotiation successful
0812 11:16:07.695455 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":194,"threads_started":1}
I20260812 11:16:07.699092 12402 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36017
I20260812 11:16:07.699584 12402 heartbeater.cc:464] Registering TS with master...
I20260812 11:16:07.700299 12402 heartbeater.cc:511] Master 127.27.109.62:36017 requested a full tablet report, sending...
I20260812 11:16:07.701797 11653 ts_manager.cc:194] Registered new tserver with Master: eea454f3e8f140499d159877ca5165cd (127.27.109.3:37805)
I20260812 11:16:07.702482 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:60473
I20260812 11:16:07.711436 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:16:07.724609 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.714507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.27.109.62:36017)
0812 11:16:07.714926 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:07.714934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:07.715062 (+ 128us) server_negotiation.cc:407] Connection header received
0812 11:16:07.715266 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:07.715270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:07.715332 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:07.715438 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:07.716415 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.717045 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.718057 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.718328 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.718459 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.720188 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:16:07.720193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:16:07.720196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:16:07.720233 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:16:07.722418 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.723073 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.723077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.723079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.723179 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:16:07.723508 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:16:07.723511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:16:07.723513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:16:07.723815 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:16:07.723981 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:07.724259 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 11:16:07.724413 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260812 11:16:07.729782 11653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:16:07.733544 11653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:16:07.755797 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.747667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44631 (local address 127.0.0.1:45386)
0812 11:16:07.748744 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:07.748761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:07.748937 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:07.749811 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:07.749818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:07.749857 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:16:07.750123 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.750134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.751938 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.751942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.753555 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.753568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.754916 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.754921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.755093 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.755149 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 11:16:07.755346 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 11:16:07.755572 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":97,"spinlock_wait_cycles":27392,"thread_start_us":185,"threads_started":1}
I20260812 11:16:07.756459 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.747571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44041 (local address 127.0.0.1:58922)
0812 11:16:07.748684 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:07.748762 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:07.748937 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:07.749811 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:07.749818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:07.749854 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:16:07.750123 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.750135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.752237 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.752245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.753575 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.753589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.755322 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.755328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.755449 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.755483 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:16:07.755577 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 11:16:07.755703 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":85,"spinlock_wait_cycles":15360,"thread_start_us":121,"threads_started":1}
I20260812 11:16:07.756703 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.747685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.27.109.2:44041)
0812 11:16:07.748611 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:07.748624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:07.749100 (+ 476us) server_negotiation.cc:407] Connection header received
0812 11:16:07.749197 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:07.749202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:07.749450 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:07.749606 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:16:07.750319 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.751999 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.753798 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.754917 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.755049 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.755239 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:07.755885 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 11:16:07.756520 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":170,"threads_started":1}
I20260812 11:16:07.756829 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.748871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.27.109.3:37805)
0812 11:16:07.749491 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:07.749506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:07.749835 (+ 329us) server_negotiation.cc:407] Connection header received
0812 11:16:07.750160 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:07.750173 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:07.750587 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:07.750988 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:16:07.751770 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.753042 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.754674 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.755606 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.755684 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.755821 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:07.756547 (+ 726us) server_negotiation.cc:299] Negotiation successful
0812 11:16:07.756658 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":250,"threads_started":1}
I20260812 11:16:07.757043 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.747924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.27.109.1:44631)
0812 11:16:07.748491 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:07.748503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:07.749119 (+ 616us) server_negotiation.cc:407] Connection header received
0812 11:16:07.749212 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:07.749217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:07.749450 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:07.749606 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:16:07.750319 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.751763 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.753773 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.754619 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.754890 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.755156 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:07.755396 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 11:16:07.755569 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":162,"threads_started":1}
I20260812 11:16:07.757252 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:07.748628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:37805 (local address 127.0.0.1:45024)
0812 11:16:07.749706 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:07.749726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:07.749952 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:07.751336 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:07.751340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:07.751391 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:16:07.751565 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.751573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.753240 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.753244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.754497 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:07.754510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:07.756154 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:07.756161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:07.756272 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:07.756299 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:16:07.756979 (+ 680us) client_negotiation.cc:240] Negotiation successful
0812 11:16:07.757089 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":1,"spinlock_wait_cycles":3200,"thread_start_us":165,"threads_started":1}
I20260812 11:16:07.760794 11860 tablet_service.cc:1467] Processing CreateTablet for tablet c933e6c44cdd451fba35c07adfb743ac (DEFAULT_TABLE table=test-table [id=cda9ff6a7cb449c686f6a875c14107cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:16:07.760794 12066 tablet_service.cc:1467] Processing CreateTablet for tablet c933e6c44cdd451fba35c07adfb743ac (DEFAULT_TABLE table=test-table [id=cda9ff6a7cb449c686f6a875c14107cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:16:07.761410 12294 tablet_service.cc:1467] Processing CreateTablet for tablet c933e6c44cdd451fba35c07adfb743ac (DEFAULT_TABLE table=test-table [id=cda9ff6a7cb449c686f6a875c14107cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:16:07.762331 12066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c933e6c44cdd451fba35c07adfb743ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:16:07.762331 11860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c933e6c44cdd451fba35c07adfb743ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:16:07.763146 12294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c933e6c44cdd451fba35c07adfb743ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:16:07.772580 12430 tablet_bootstrap.cc:492] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba: Bootstrap starting.
I20260812 11:16:07.773689 12434 tablet_bootstrap.cc:492] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Bootstrap starting.
I20260812 11:16:07.774351 12431 tablet_bootstrap.cc:492] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Bootstrap starting.
I20260812 11:16:07.776793 12430 tablet_bootstrap.cc:654] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba: Neither blocks nor log segments found. Creating new log.
I20260812 11:16:07.777681 12431 tablet_bootstrap.cc:654] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Neither blocks nor log segments found. Creating new log.
I20260812 11:16:07.777976 12434 tablet_bootstrap.cc:654] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Neither blocks nor log segments found. Creating new log.
I20260812 11:16:07.778263 12430 log.cc:824] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba: Log is configured to *not* fsync() on all Append() calls
I20260812 11:16:07.779443 12434 log.cc:824] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:16:07.779443 12431 log.cc:824] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:16:07.781566 12430 tablet_bootstrap.cc:492] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba: No bootstrap required, opened a new log
I20260812 11:16:07.781876 12430 ts_tablet_manager.cc:1397] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260812 11:16:07.782531 12434 tablet_bootstrap.cc:492] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: No bootstrap required, opened a new log
I20260812 11:16:07.782881 12434 ts_tablet_manager.cc:1397] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260812 11:16:07.782893 12431 tablet_bootstrap.cc:492] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: No bootstrap required, opened a new log
I20260812 11:16:07.783246 12431 ts_tablet_manager.cc:1397] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 11:16:07.786927 12434 raft_consensus.cc:348] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:07.787348 12434 raft_consensus.cc:374] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:16:07.787417 12434 raft_consensus.cc:729] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830401feab7245d2a491acdac788c4d0, State: Initialized, Role: FOLLOWER
I20260812 11:16:07.787611 12430 raft_consensus.cc:348] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:07.787823 12431 raft_consensus.cc:348] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:07.788092 12430 raft_consensus.cc:374] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:16:07.788091 12431 raft_consensus.cc:374] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:16:07.788167 12430 raft_consensus.cc:729] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51027dcac654d8d8de2a694292fe4ba, State: Initialized, Role: FOLLOWER
I20260812 11:16:07.788167 12431 raft_consensus.cc:729] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea454f3e8f140499d159877ca5165cd, State: Initialized, Role: FOLLOWER
I20260812 11:16:07.788144 12434 consensus_queue.cc:260] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:07.788759 12431 consensus_queue.cc:260] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:07.788759 12430 consensus_queue.cc:260] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:07.789459 12194 heartbeater.cc:503] Master 127.27.109.62:36017 was elected leader, sending a full tablet report...
I20260812 11:16:07.790126 12430 ts_tablet_manager.cc:1428] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260812 11:16:07.790126 12431 ts_tablet_manager.cc:1428] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 11:16:07.790182 11962 heartbeater.cc:503] Master 127.27.109.62:36017 was elected leader, sending a full tablet report...
I20260812 11:16:07.791044 12434 ts_tablet_manager.cc:1428] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:16:07.791929 12402 heartbeater.cc:503] Master 127.27.109.62:36017 was elected leader, sending a full tablet report...
W20260812 11:16:07.804754 11963 tablet.cc:2369] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:16:07.882375 12195 tablet.cc:2369] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:16:07.923493 12404 tablet.cc:2369] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:16:08.064620 12443 raft_consensus.cc:482] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:16:08.064949 12443 raft_consensus.cc:504] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:08.066285 12443 leader_election.cc:288] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830401feab7245d2a491acdac788c4d0 (127.27.109.2:44041), eea454f3e8f140499d159877ca5165cd (127.27.109.3:37805)
I20260812 11:16:08.073601 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:08.067807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:37517 (local address 127.27.109.3:37805)
0812 11:16:08.067963 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:08.067970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:08.067991 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:16:08.068236 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:08.068238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:08.068280 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:08.068355 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:16:08.068656 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.069389 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.071884 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.072667 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.072827 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:08.072898 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:08.073332 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 11:16:08.073436 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:16:08.073601 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:08.067083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:56249 (local address 127.27.109.2:44041)
0812 11:16:08.067242 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:08.067246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:08.067778 (+ 532us) server_negotiation.cc:407] Connection header received
0812 11:16:08.067857 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:08.067861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:08.067929 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:08.068045 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:16:08.068478 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.069316 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.071910 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.072644 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.072827 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:08.072900 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:08.073332 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 11:16:08.073436 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:16:08.073666 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:08.067615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:37805 (local address 127.27.109.1:37517)
0812 11:16:08.067927 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:08.067940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:08.068090 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:08.068412 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:08.068415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:08.068438 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:16:08.068535 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:08.068540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.069548 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.069552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:08.071656 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:08.071673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.072865 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.072870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:08.073080 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:08.073128 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 11:16:08.073244 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:16:08.073373 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":219,"thread_start_us":121,"threads_started":1}
I20260812 11:16:08.073670 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:08.066970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44041 (local address 127.27.109.1:56249)
0812 11:16:08.067437 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:08.067459 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:16:08.067645 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:16:08.068090 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:16:08.068094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:16:08.068110 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:16:08.068297 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:08.068303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.069612 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.069615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:08.071664 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:16:08.071673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.072865 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.072870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:16:08.073073 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:08.073132 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 11:16:08.073243 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:16:08.073373 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":365,"thread_start_us":210,"threads_started":1}
I20260812 11:16:08.075079 12096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c933e6c44cdd451fba35c07adfb743ac" candidate_uuid: "b51027dcac654d8d8de2a694292fe4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830401feab7245d2a491acdac788c4d0" is_pre_election: true
I20260812 11:16:08.075101 12327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c933e6c44cdd451fba35c07adfb743ac" candidate_uuid: "b51027dcac654d8d8de2a694292fe4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea454f3e8f140499d159877ca5165cd" is_pre_election: true
I20260812 11:16:08.075496 12096 raft_consensus.cc:2454] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51027dcac654d8d8de2a694292fe4ba in term 0.
I20260812 11:16:08.075515 12327 raft_consensus.cc:2454] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51027dcac654d8d8de2a694292fe4ba in term 0.
I20260812 11:16:08.076131 11789 leader_election.cc:302] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830401feab7245d2a491acdac788c4d0, b51027dcac654d8d8de2a694292fe4ba; no voters:
I20260812 11:16:08.076540 12443 raft_consensus.cc:2781] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:16:08.076665 12443 raft_consensus.cc:482] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:16:08.076699 12443 raft_consensus.cc:3037] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:16:08.077970 12443 raft_consensus.cc:504] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:08.078509 12443 leader_election.cc:288] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [CANDIDATE]: Term 1 election: Requested vote from peers 830401feab7245d2a491acdac788c4d0 (127.27.109.2:44041), eea454f3e8f140499d159877ca5165cd (127.27.109.3:37805)
I20260812 11:16:08.079309 12327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c933e6c44cdd451fba35c07adfb743ac" candidate_uuid: "b51027dcac654d8d8de2a694292fe4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea454f3e8f140499d159877ca5165cd"
I20260812 11:16:08.079335 12096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c933e6c44cdd451fba35c07adfb743ac" candidate_uuid: "b51027dcac654d8d8de2a694292fe4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830401feab7245d2a491acdac788c4d0"
I20260812 11:16:08.079509 12327 raft_consensus.cc:3037] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:16:08.079535 12096 raft_consensus.cc:3037] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:16:08.081184 12327 raft_consensus.cc:2454] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51027dcac654d8d8de2a694292fe4ba in term 1.
I20260812 11:16:08.081197 12096 raft_consensus.cc:2454] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51027dcac654d8d8de2a694292fe4ba in term 1.
I20260812 11:16:08.081626 11789 leader_election.cc:302] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830401feab7245d2a491acdac788c4d0, b51027dcac654d8d8de2a694292fe4ba; no voters:
I20260812 11:16:08.081980 12443 raft_consensus.cc:2781] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:16:08.082552 12443 raft_consensus.cc:686] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [term 1 LEADER]: Becoming Leader. State: Replica: b51027dcac654d8d8de2a694292fe4ba, State: Running, Role: LEADER
I20260812 11:16:08.083076 12443 consensus_queue.cc:237] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } }
I20260812 11:16:08.087924 11651 catalog_manager.cc:5495] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba reported cstate change: term changed from 0 to 1, leader changed from <none> to b51027dcac654d8d8de2a694292fe4ba (127.27.109.1). New cstate: current_term: 1 leader_uuid: "b51027dcac654d8d8de2a694292fe4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51027dcac654d8d8de2a694292fe4ba" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44631 } health_report { overall_health: HEALTHY } } }
I20260812 11:16:08.141945 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:08.134492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.27.109.1:44631)
0812 11:16:08.134673 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:08.134678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:08.134697 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:16:08.135080 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:08.135083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:08.135168 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:08.135294 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:16:08.135779 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.136455 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.138464 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:08.138904 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:08.139061 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:08.139868 (+ 807us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:16:08.140834 (+ 966us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:16:08.141010 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:08.141549 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 11:16:08.141649 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:16:08.152235 12096 raft_consensus.cc:1261] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 1 FOLLOWER]: Refusing update from remote peer b51027dcac654d8d8de2a694292fe4ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:16:08.152235 12327 raft_consensus.cc:1261] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 1 FOLLOWER]: Refusing update from remote peer b51027dcac654d8d8de2a694292fe4ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:16:08.154263 12448 consensus_queue.cc:1035] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "830401feab7245d2a491acdac788c4d0" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:16:08.154692 12443 consensus_queue.cc:1035] T c933e6c44cdd451fba35c07adfb743ac P b51027dcac654d8d8de2a694292fe4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea454f3e8f140499d159877ca5165cd" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:16:08.181784 11650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:16:08.182060 11650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:16:08.187942 11650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7de7421c15a04f65a0aeac2b161056b8: Sending DeleteTablet for 3 replicas of tablet c933e6c44cdd451fba35c07adfb743ac
I20260812 11:16:08.190213 12294 tablet_service.cc:1514] Processing DeleteTablet for tablet c933e6c44cdd451fba35c07adfb743ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:16:08 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45024
I20260812 11:16:08.190192 12066 tablet_service.cc:1514] Processing DeleteTablet for tablet c933e6c44cdd451fba35c07adfb743ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:16:08 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:58922
I20260812 11:16:08.190948 11860 tablet_service.cc:1514] Processing DeleteTablet for tablet c933e6c44cdd451fba35c07adfb743ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:16:08 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45386
I20260812 11:16:08.191438 12469 tablet_replica.cc:330] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: stopping tablet replica
I20260812 11:16:08.191521 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 11753
I20260812 11:16:08.192049 12469 raft_consensus.cc:2229] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:16:08.192575 12469 raft_consensus.cc:2258] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:16:08.191706 12471 tablet_replica.cc:330] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: stopping tablet replica
I20260812 11:16:08.215837 12471 raft_consensus.cc:2229] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:16:08.216363 12471 raft_consensus.cc:2258] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:16:08.216496 12469 ts_tablet_manager.cc:1905] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:16:08.220520 12469 ts_tablet_manager.cc:1918] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:16:08.220710 12469 log.cc:1196] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/c933e6c44cdd451fba35c07adfb743ac
I20260812 11:16:08.221098 12469 ts_tablet_manager.cc:1939] T c933e6c44cdd451fba35c07adfb743ac P 830401feab7245d2a491acdac788c4d0: Deleting consensus metadata
I20260812 11:16:08.224892 11632 catalog_manager.cc:4841] TS 830401feab7245d2a491acdac788c4d0 (127.27.109.2:44041): tablet c933e6c44cdd451fba35c07adfb743ac (table test-table [id=cda9ff6a7cb449c686f6a875c14107cc]) successfully deleted
I20260812 11:16:08.225062 12471 ts_tablet_manager.cc:1905] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:16:08.228330 12471 ts_tablet_manager.cc:1918] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:16:08.228533 12471 log.cc:1196] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/c933e6c44cdd451fba35c07adfb743ac
I20260812 11:16:08.228937 12471 ts_tablet_manager.cc:1939] T c933e6c44cdd451fba35c07adfb743ac P eea454f3e8f140499d159877ca5165cd: Deleting consensus metadata
I20260812 11:16:08.230458 11632 catalog_manager.cc:4841] TS eea454f3e8f140499d159877ca5165cd (127.27.109.3:37805): tablet c933e6c44cdd451fba35c07adfb743ac (table test-table [id=cda9ff6a7cb449c686f6a875c14107cc]) successfully deleted
W20260812 11:16:08.233345 11628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:44631 (error 108)
I20260812 11:16:08.233803 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 11971
I20260812 11:16:08.240711 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:08.239858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44631 (local address 127.0.0.1:45408)
0812 11:16:08.240390 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 11:16:08.240536 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:44631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
W20260812 11:16:08.241309 11628 catalog_manager.cc:4584] TS b51027dcac654d8d8de2a694292fe4ba (127.27.109.1:44631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c933e6c44cdd451fba35c07adfb743ac: Network error: Client connection negotiation failed: client connection to 127.27.109.1:44631: connect: Connection refused (error 111)
I20260812 11:16:08.257506 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 12200
I20260812 11:16:08.276822 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 11598
2026-08-12T11:16:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3640 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:16:08 dist-test-slave-7z32 krb5kdc[12481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:16:08 dist-test-slave-7z32 krb5kdc[12481](info): set up 2 sockets
Aug 12 11:16:08 dist-test-slave-7z32 krb5kdc[12481](info): commencing operation
krb5kdc: starting...
W20260812 11:16:09.656302 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.338s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:16:09 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:16:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
I20260812 11:16:09.903887 28084 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskK9g2P8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/postgres -l logfile start
2026-08-12 11:16:13.560 UTC [12530] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 11:16:13.561 UTC [12530] LOG: listening on IPv4 address "127.27.109.20", port 46971
2026-08-12 11:16:13.565 UTC [12530] LOG: listening on Unix socket "/tmp/.s.PGSQL.46971"
2026-08-12 11:16:13.596 UTC [12533] LOG: database system was shut down at 2026-08-12 11:16:11 UTC
2026-08-12 11:16:13.600 UTC [12530] LOG: database system is ready to accept connections
I20260812 11:16:14.851467 28084 mini_postgres.cc:96] Postgres bound to 46971
2026-08-12 11:16:14.862 UTC [12548] FATAL: database "slave" does not exist
127.27.109.20:46971 - accepting connections
I20260812 11:16:14.864164 28084 mini_ranger.cc:161] Starting Ranger...
I20260812 11:16:14.885546 28084 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 11:16:15.673756 28084 mini_ranger.cc:87] Created ranger Postgres database
I20260812 11:16:15.673888 28084 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-admin
2026-08-12 11:16:16,093 [I] DB FLAVOR :POSTGRES
2026-08-12 11:16:16,094 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:16:16,094 [I] Checking connection..
2026-08-12 11:16:16,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 11:16:16,456 [I] Checking connection passed.
2026-08-12 11:16:16,457 [I] --------- Verifying version history table ---------
2026-08-12 11:16:16,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:16:16,783 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:16:16,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:16:17,149 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:16:17,149 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 11:16:17,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 11:16:17.485 UTC [12651] WARNING: there is no transaction in progress
2026-08-12 11:16:17,494 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 11:16:17,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:16:17,821 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 11:16:17,822 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:16:17,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 11:16:18,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 11:16:18,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7z32.c.gcp-upstream.internal','N') ;"
2026-08-12 11:16:18,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:16:19,131 [I] Table x_portal_user does not exist in database ranger
2026-08-12 11:16:19,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:16:19,459 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 11:16:19,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:16:19,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:16:20,085 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 11:16:20,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 11:16:21.416 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.439 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.465 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.491 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.519 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.548 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.639 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.649 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.672 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.686 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.702 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.721 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.735 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.745 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.756 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:21.767 UTC [12814] WARNING: there is no transaction in progress
2026-08-12 11:16:22,858 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 11:16:22,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 11:16:23,166 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 11:16:23,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:16:23,510 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 11:16:23,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:16:23,840 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 11:16:23,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:16:24,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:16:24,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7z32.c.gcp-upstream.internal';"
2026-08-12 11:16:24,797 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 11:16:24,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 11:16:25,110 [I] DB_PATCHES have already been applied
I20260812 11:16:25.118479 28084 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-admin:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskK9g2P8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 11:16:25.118580 28084 mini_ranger.cc:193] Using host: 127.27.109.20
I20260812 11:16:25.125144 28084 mini_ranger.cc:227] Ranger admin URL: http://127.27.109.20:42581
Aug 12, 2026 11:16:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:16:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:16:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 11:16:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 11:16:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 11:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 11:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 11:16:26.749918 28084 mini_ranger.cc:160] Time spent starting Ranger: real 11.886s user 0.003s sys 0.017s
Aug 12, 2026 11:16:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 11:16:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 11:16:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 11:16:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 11:16:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 11:16:53.699541 28084 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 11:16:53 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533413, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 11:16:53.831833 28084 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 11:16:53.848026 28084 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 11:16:54.947108 28084 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 11:16:54.947235 28084 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms
I20260812 11:16:54.947453 28084 mini_ranger_kms.cc:227] Using postgres at 127.27.109.20:46971
2026-08-12 11:16:55,637 [I] DB FLAVOR :POSTGRES
2026-08-12 11:16:55,637 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:16:55,638 [I] Checking connection
2026-08-12 11:16:55,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 11:16:55,972 [I] connection success
2026-08-12 11:16:55,973 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 11:16:55,973 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 11:16:55,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 11:16:56,303 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 11:16:56,304 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:16:56,304 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 11:16:56,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:46971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 11:16:56,652 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 11:16:57.359541 28084 mini_ranger_kms.cc:310] Created kms service
I20260812 11:16:57.589607 28084 mini_ranger_kms.cc:326] Created kudu user
I20260812 11:16:57.642865 28084 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 11:16:57 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533413, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:16:58.230176 28084 mini_ranger_kms.cc:384] Added ranger policy
I20260812 11:16:58.230427 28084 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms:/tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 11:16:58.230454 28084 mini_ranger_kms.cc:240] Using host: 127.27.109.20
I20260812 11:16:58.239382 28084 mini_ranger_kms.cc:276] Ranger KMS PID: 13085
I20260812 11:16:58.239729 28084 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.109.20:51709
Aug 12, 2026 11:16:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:16:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:16:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 11:16:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 11:17:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 11:17:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 11:17:01 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:17:01.366567 28084 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.535s user 0.010s sys 0.351s
I20260812 11:17:01.366746 28084 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 11:17:01.366942 28084 mini_ranger_kms.cc:401] 127.27.109.20:51709/kms/v1/keys
Aug 12 11:17:01 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 12 11:17:02 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 12 11:17:05 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:17:05.295923 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45637
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45637
--ranger_config_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:51709/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:17:05.430449 13164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:17:05.430857 13164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:17:05.430939 13164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:17:05.435231 13164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:17:05.435338 13164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:17:05.435374 13164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:17:05.435415 13164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:17:05.435453 13164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:17:05.441339 13164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.109.20:51709/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45637
--ranger_config_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:17:05.442577 13164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:17:05.443732 13164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:17:05.454169 13173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:05.454212 13171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:05.454169 13170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:17:05.454524 13164 server_base.cc:1034] running on GCE node
I20260812 11:17:05.455608 13164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:17:05.457310 13164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:17:05.458621 13164 hybrid_clock.cc:648] HybridClock initialized: now 1786533425458574 us; error 64 us; skew 500 ppm
Aug 12 11:17:05 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533425, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:17:05.463510 13164 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:17:05.465251 13164 webserver.cc:460] Webserver started at http://127.27.109.62:45323/ using document root <none> and password file <none>
I20260812 11:17:05.466120 13164 fs_manager.cc:359] Metadata directory not provided
I20260812 11:17:05.466245 13164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:17:05.466571 13164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:17:05 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533425, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:17:05.535866 13164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "27184c740f83474bbcf4a4603183f0e2"
format_stamp: "Formatted at 2026-08-12 11:17:05 on dist-test-slave-7z32"
server_key: "90b7c3276d24e521932fbb53e87a8a2b"
server_key_iv: "2e2e42a0279989c50319f4d0e983e29f"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:05.536724 13164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "27184c740f83474bbcf4a4603183f0e2"
format_stamp: "Formatted at 2026-08-12 11:17:05 on dist-test-slave-7z32"
server_key: "90b7c3276d24e521932fbb53e87a8a2b"
server_key_iv: "2e2e42a0279989c50319f4d0e983e29f"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:05.542313 13164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 11:17:05.564344 13183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:05.566977 13164 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260812 11:17:05.567260 13164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "27184c740f83474bbcf4a4603183f0e2"
format_stamp: "Formatted at 2026-08-12 11:17:05 on dist-test-slave-7z32"
server_key: "90b7c3276d24e521932fbb53e87a8a2b"
server_key_iv: "2e2e42a0279989c50319f4d0e983e29f"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:05.567464 13164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:17:05.614586 13164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:17:05.618739 13164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:17:05.619412 13164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:17:05.628736 13164 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:45637
I20260812 11:17:05.628907 13266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:45637 every 8 connection(s)
I20260812 11:17:05.630028 13164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:17:05.634258 13272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:17:05.640069 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 13164
I20260812 11:17:05.640195 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:17:05.640558 28084 external_mini_cluster.cc:1428] Setting key ba9de90d470ecf0bb9059179c250a001
I20260812 11:17:05.643368 13272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2: Bootstrap starting.
I20260812 11:17:05.647626 13272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2: Neither blocks nor log segments found. Creating new log.
I20260812 11:17:05.649026 13272 log.cc:824] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:17:05.652539 13272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2: No bootstrap required, opened a new log
Aug 12 11:17:05 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:17:05.658545 13272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27184c740f83474bbcf4a4603183f0e2" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45637 } }
I20260812 11:17:05.659061 13272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:17:05.659129 13272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27184c740f83474bbcf4a4603183f0e2, State: Initialized, Role: FOLLOWER
I20260812 11:17:05.659894 13272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27184c740f83474bbcf4a4603183f0e2" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45637 } }
I20260812 11:17:05.660152 13272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:17:05.660243 13272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:17:05.660373 13272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:17:05.662230 13272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27184c740f83474bbcf4a4603183f0e2" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45637 } }
I20260812 11:17:05.662765 13272 leader_election.cc:302] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27184c740f83474bbcf4a4603183f0e2; no voters:
I20260812 11:17:05.663347 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:05.642497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.27.109.62:45637)
0812 11:17:05.643090 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:05.643110 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:05.643279 (+ 169us) server_negotiation.cc:407] Connection header received
0812 11:17:05.644512 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:05.644538 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:05.644954 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:05.645321 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:17:05.646774 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:05.648372 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:05.649969 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:05.650932 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:05.651170 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:05.655283 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:17:05.655299 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:17:05.655332 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:17:05.655388 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:17:05.658099 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:05.658914 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:05.658924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:05.658932 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:05.659131 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:05.659734 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:05.659738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:05.659740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:05.660547 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:17:05.660817 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:05.661191 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 11:17:05.661631 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":192,"threads_started":1}
I20260812 11:17:05.663544 13272 leader_election.cc:288] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:17:05.664001 13284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:17:05.664418 13284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [term 1 LEADER]: Becoming Leader. State: Replica: 27184c740f83474bbcf4a4603183f0e2, State: Running, Role: LEADER
I20260812 11:17:05.664901 13284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27184c740f83474bbcf4a4603183f0e2" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45637 } }
I20260812 11:17:05.665550 13272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:17:05.667145 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27184c740f83474bbcf4a4603183f0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27184c740f83474bbcf4a4603183f0e2" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45637 } } }
I20260812 11:17:05.667312 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [sys.catalog]: This master's current role is: LEADER
I20260812 11:17:05.667725 13286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27184c740f83474bbcf4a4603183f0e2. Latest consensus state: current_term: 1 leader_uuid: "27184c740f83474bbcf4a4603183f0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27184c740f83474bbcf4a4603183f0e2" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45637 } } }
I20260812 11:17:05.667948 13286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2 [sys.catalog]: This master's current role is: LEADER
I20260812 11:17:05.667991 13291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:17:05.673553 13291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:17:05.674401 13272 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:17:05.688864 13291 catalog_manager.cc:1347] Generated new cluster ID: f81225268eba47eba3d55d4944d51d79
I20260812 11:17:05.689023 13291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:17:05.712488 13291 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:17:05.714769 13291 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:17:05.724360 13291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2: Generated new TSK 0
I20260812 11:17:05.727029 13291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:17:06.556023 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:51709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:45637
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:17:06.754527 13347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:17:06.755111 13347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:17:06.755247 13347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:17:06.760291 13347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:17:06.760433 13347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:17:06.760573 13347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:17:06.767629 13347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.109.20:51709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:17:06.769661 13347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:17:06.771560 13347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:17:06.785290 13358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:06.785184 13359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:06.785198 13361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:17:06.787221 13347 server_base.cc:1034] running on GCE node
I20260812 11:17:06.788069 13347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:17:06.789122 13347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:17:06.790369 13347 hybrid_clock.cc:648] HybridClock initialized: now 1786533426790327 us; error 64 us; skew 500 ppm
Aug 12 11:17:06 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533426, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:17:06.794734 13347 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:17:06.796243 13347 webserver.cc:460] Webserver started at http://127.27.109.1:45239/ using document root <none> and password file <none>
I20260812 11:17:06.797398 13347 fs_manager.cc:359] Metadata directory not provided
I20260812 11:17:06.797554 13347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:17:06.798430 13347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:17:06 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533426, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:17:06.826884 13347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "5d207bc5103d48aeb46f653f4619d0ca"
format_stamp: "Formatted at 2026-08-12 11:17:06 on dist-test-slave-7z32"
server_key: "5d692163b53613f49343d53d315f9e31"
server_key_iv: "66be92c684ea1a6fe54dcaddfcf77aab"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:06.827968 13347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "5d207bc5103d48aeb46f653f4619d0ca"
format_stamp: "Formatted at 2026-08-12 11:17:06 on dist-test-slave-7z32"
server_key: "5d692163b53613f49343d53d315f9e31"
server_key_iv: "66be92c684ea1a6fe54dcaddfcf77aab"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:06.841005 13347 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260812 11:17:06.872572 13374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:06.875445 13347 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260812 11:17:06.875710 13347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "5d207bc5103d48aeb46f653f4619d0ca"
format_stamp: "Formatted at 2026-08-12 11:17:06 on dist-test-slave-7z32"
server_key: "5d692163b53613f49343d53d315f9e31"
server_key_iv: "66be92c684ea1a6fe54dcaddfcf77aab"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:06.875922 13347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:17:06.907899 13347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:17:06.913159 13347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:17:06.913514 13347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:17:06.914475 13347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:17:06.915921 13347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:17:06.916044 13347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 11:17:06.916216 13347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:17:06.916311 13347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:06.935478 13347 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:38419
I20260812 11:17:06.935498 13566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:38419 every 8 connection(s)
I20260812 11:17:06.937202 13347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:17:06.941496 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 13347
I20260812 11:17:06.941695 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:17:06.942085 28084 external_mini_cluster.cc:1428] Setting key 77430b499f1c39deb969ff171b75b41b
Aug 12 11:17:06 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533426, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:17:06.967228 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:06.940225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:37479 (local address 127.27.109.62:45637)
0812 11:17:06.940558 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:06.940564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:06.943769 (+ 3205us) server_negotiation.cc:407] Connection header received
0812 11:17:06.949012 (+ 5243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:06.949019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:06.949105 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:06.949179 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:17:06.951661 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:06.952681 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:06.954794 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:06.955305 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:06.955463 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:06.959976 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:17:06.959986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:17:06.959991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:17:06.960061 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:17:06.962467 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:06.963713 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:06.963720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:06.963722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:06.963976 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:06.964706 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:06.964711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:06.964713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:06.965333 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:17:06.965458 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:06.966713 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0812 11:17:06.967011 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:17:06.969184 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:06.940938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45637 (local address 127.27.109.1:37479)
0812 11:17:06.943525 (+ 2587us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:06.943621 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:06.948524 (+ 4903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:06.949739 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:06.949751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:06.950291 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:17:06.951361 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:06.951394 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:06.952929 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:06.952936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:06.954592 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:06.954609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:06.955787 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:06.955795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:06.956161 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:06.957154 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:17:06.957181 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:17:06.959563 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:17:06.962715 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:17:06.962728 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:17:06.962778 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:17:06.963381 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:17:06.964166 (+ 785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:17:06.964170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:17:06.964173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:17:06.964517 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:17:06.965573 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:17:06.965585 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:17:06.966140 (+ 555us) client_negotiation.cc:769] Sending connection context
0812 11:17:06.966497 (+ 357us) client_negotiation.cc:240] Negotiation successful
0812 11:17:06.967130 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2242,"thread_start_us":228,"threads_started":1}
I20260812 11:17:06.972110 13567 heartbeater.cc:347] Connected to a master server at 127.27.109.62:45637
I20260812 11:17:06.972590 13567 heartbeater.cc:464] Registering TS with master...
I20260812 11:17:06.973618 13567 heartbeater.cc:511] Master 127.27.109.62:45637 requested a full tablet report, sending...
I20260812 11:17:06.977020 13216 ts_manager.cc:194] Registered new tserver with Master: 5d207bc5103d48aeb46f653f4619d0ca (127.27.109.1:38419)
I20260812 11:17:06.979089 13216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:37479
Aug 12 11:17:06 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533426, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:17:06.999624 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:51709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:45637
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:17:07.162775 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:17:07.163187 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:17:07.163233 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:17:07.168996 13587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:17:07.169106 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:17:07.169209 13587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:17:07.176071 13587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.109.20:51709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:17:07.177464 13587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:17:07.178730 13587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:17:07.192193 13595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:07.192198 13596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:07.192763 13601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:17:07.194092 13587 server_base.cc:1034] running on GCE node
I20260812 11:17:07.194717 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:17:07.195592 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:17:07.196895 13587 hybrid_clock.cc:648] HybridClock initialized: now 1786533427196855 us; error 50 us; skew 500 ppm
Aug 12 11:17:07 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533427, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:17:07.201253 13587 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:17:07.202813 13587 webserver.cc:460] Webserver started at http://127.27.109.2:40861/ using document root <none> and password file <none>
I20260812 11:17:07.203954 13587 fs_manager.cc:359] Metadata directory not provided
I20260812 11:17:07.204061 13587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:17:07.204357 13587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:17:07 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533427, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:17:07.234025 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a"
format_stamp: "Formatted at 2026-08-12 11:17:07 on dist-test-slave-7z32"
server_key: "2c81603c8e022d65e665684b74792f4b"
server_key_iv: "e13f5c4b97cd5222e15fab2e31284372"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:07.234943 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a"
format_stamp: "Formatted at 2026-08-12 11:17:07 on dist-test-slave-7z32"
server_key: "2c81603c8e022d65e665684b74792f4b"
server_key_iv: "e13f5c4b97cd5222e15fab2e31284372"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:07.240722 13587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 11:17:07.263379 13615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:07.265527 13587 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260812 11:17:07.265681 13587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a"
format_stamp: "Formatted at 2026-08-12 11:17:07 on dist-test-slave-7z32"
server_key: "2c81603c8e022d65e665684b74792f4b"
server_key_iv: "e13f5c4b97cd5222e15fab2e31284372"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:07.265903 13587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:17:07.284556 13587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:17:07.289638 13587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:17:07.289877 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:17:07.290690 13587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:17:07.292101 13587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:17:07.292181 13587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:07.292240 13587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:17:07.292281 13587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:07.307628 13587 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43455
I20260812 11:17:07.308971 13587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:17:07.316205 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 13587
I20260812 11:17:07.316386 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:17:07.316784 28084 external_mini_cluster.cc:1428] Setting key 06ab4a16a428074fcc4f42615e530561
I20260812 11:17:07.317837 13803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43455 every 8 connection(s)
Aug 12 11:17:07 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533427, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:17:07.333423 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.314524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:49709 (local address 127.27.109.62:45637)
0812 11:17:07.314675 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.314679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.314699 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:17:07.316050 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.316056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.316141 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.316209 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:17:07.318990 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.319890 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.321994 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.322440 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.322595 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.326780 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:17:07.326801 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:17:07.326804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:17:07.326898 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:17:07.329959 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:07.331113 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:07.331121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:07.331124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:07.331244 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:07.331764 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:07.331769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:07.331774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:07.332068 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:17:07.332166 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.333031 (+ 865us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.333205 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:17:07.335731 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.313081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45637 (local address 127.27.109.2:49709)
0812 11:17:07.313972 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.314072 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.315625 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.316823 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.316836 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.317367 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:17:07.318665 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.318696 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.320106 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.320110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.321780 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.321797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.322907 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.322915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.323210 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.324165 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:17:07.324192 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:17:07.326448 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:17:07.330354 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:17:07.330364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:17:07.330390 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:17:07.330939 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:17:07.331378 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:17:07.331384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:17:07.331386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:17:07.331634 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:17:07.332249 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:17:07.332260 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:17:07.332700 (+ 440us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.334012 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.334472 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":221,"threads_started":1}
I20260812 11:17:07.338091 13804 heartbeater.cc:347] Connected to a master server at 127.27.109.62:45637
I20260812 11:17:07.338438 13804 heartbeater.cc:464] Registering TS with master...
I20260812 11:17:07.339247 13804 heartbeater.cc:511] Master 127.27.109.62:45637 requested a full tablet report, sending...
I20260812 11:17:07.340780 13216 ts_manager.cc:194] Registered new tserver with Master: d3b7bb664e5f4a41a3402fa1f9b7437a (127.27.109.2:43455)
I20260812 11:17:07.341642 13216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:49709
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:17:07.383222 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:51709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:45637
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:17:07.550294 13814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:17:07.550676 13814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:17:07.550776 13814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:17:07.556052 13814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:17:07.556171 13814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:17:07.556298 13814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:17:07.564512 13814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.109.20:51709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:17:07.566250 13814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:17:07.567868 13814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:17:07.585263 13825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:07.586411 13837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:17:07.586668 13826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:17:07.588920 13814 server_base.cc:1034] running on GCE node
I20260812 11:17:07.589689 13814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:17:07.590698 13814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:17:07.595140 13814 hybrid_clock.cc:648] HybridClock initialized: now 1786533427594956 us; error 192 us; skew 500 ppm
Aug 12 11:17:07 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533426, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 12 11:17:07 dist-test-slave-7z32 krb5kdc[12481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533427, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:17:07.601168 13814 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:17:07.602998 13814 webserver.cc:460] Webserver started at http://127.27.109.3:39401/ using document root <none> and password file <none>
I20260812 11:17:07.604205 13814 fs_manager.cc:359] Metadata directory not provided
I20260812 11:17:07.604341 13814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:17:07.604764 13814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:17:07 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533427, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:17:07.626199 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "3acfca20a8784973a3494c14309558f5"
format_stamp: "Formatted at 2026-08-12 11:17:07 on dist-test-slave-7z32"
server_key: "8d6cb1179168f9d65d4fed501aea58f0"
server_key_iv: "13ba0caa7006cab6d9913759ca0e446e"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:07.627343 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "3acfca20a8784973a3494c14309558f5"
format_stamp: "Formatted at 2026-08-12 11:17:07 on dist-test-slave-7z32"
server_key: "8d6cb1179168f9d65d4fed501aea58f0"
server_key_iv: "13ba0caa7006cab6d9913759ca0e446e"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:07.643725 13814 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.003s sys 0.003s
I20260812 11:17:07.666998 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:07.669046 13814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:17:07.669245 13814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "3acfca20a8784973a3494c14309558f5"
format_stamp: "Formatted at 2026-08-12 11:17:07 on dist-test-slave-7z32"
server_key: "8d6cb1179168f9d65d4fed501aea58f0"
server_key_iv: "13ba0caa7006cab6d9913759ca0e446e"
server_key_version: "kuduclusterkey@0"
I20260812 11:17:07.669438 13814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:17:07.705687 13814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:17:07.709930 13814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:17:07.710203 13814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:17:07.711098 13814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:17:07.712473 13814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:17:07.712559 13814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:07.712637 13814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:17:07.712675 13814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:17:07.726083 13814 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:32823
I20260812 11:17:07.726104 14032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:32823 every 8 connection(s)
I20260812 11:17:07.727499 13814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:17:07.737640 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 13814
I20260812 11:17:07.737834 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:17:07.738245 28084 external_mini_cluster.cc:1428] Setting key a7469b3dbb42d3fc7765c77a30c072da
Aug 12 11:17:07 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533427, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:17:07.747189 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.731668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48443 (local address 127.27.109.62:45637)
0812 11:17:07.731827 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.731832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.731853 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:17:07.732548 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.732552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.732620 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.732758 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:17:07.734205 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.734937 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.736397 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.736667 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.736804 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.740722 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:17:07.740728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:17:07.740731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:17:07.740769 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:17:07.743022 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:07.744012 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:07.744018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:07.744020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:07.744214 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:07.745360 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:07.745364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:07.745366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:07.745705 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:17:07.745836 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.746742 (+ 906us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.746990 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:17:07.748409 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.730308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45637 (local address 127.27.109.3:48443)
0812 11:17:07.731009 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.731088 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.732262 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.732991 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.733000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.733355 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:17:07.733998 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.734017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.735089 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.735093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.736230 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.736243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.736921 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.736929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.737213 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.738223 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:17:07.738265 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:17:07.740433 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:17:07.743257 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:17:07.743266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:17:07.743290 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:17:07.743816 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:17:07.744927 (+ 1111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:17:07.744932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:17:07.744934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:17:07.745204 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:17:07.745885 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:17:07.745896 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:17:07.746366 (+ 470us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.746660 (+ 294us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.747149 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":192,"threads_started":1}
I20260812 11:17:07.750206 14034 heartbeater.cc:347] Connected to a master server at 127.27.109.62:45637
I20260812 11:17:07.750550 14034 heartbeater.cc:464] Registering TS with master...
I20260812 11:17:07.751344 14034 heartbeater.cc:511] Master 127.27.109.62:45637 requested a full tablet report, sending...
I20260812 11:17:07.753064 13216 ts_manager.cc:194] Registered new tserver with Master: 3acfca20a8784973a3494c14309558f5 (127.27.109.3:32823)
I20260812 11:17:07.753966 13216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:48443
I20260812 11:17:07.754515 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:17:07.770326 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.757808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.27.109.62:45637)
0812 11:17:07.757952 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.757957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.758504 (+ 547us) server_negotiation.cc:407] Connection header received
0812 11:17:07.758663 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.758667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.758743 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.758918 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:17:07.759937 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.760800 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.762219 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.762601 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.762745 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.765039 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:17:07.765043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:17:07.765045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:17:07.765088 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:17:07.767532 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:07.768359 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:07.768362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:07.768364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:07.768477 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:17:07.768862 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:17:07.768865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:17:07.768866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:17:07.769227 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:17:07.769362 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.769826 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.770126 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:17:07.775789 13216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:17:07.780339 13216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:17:07.809011 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.796917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43455 (local address 127.0.0.1:48574)
0812 11:17:07.797704 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.797718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.801617 (+ 3899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.803100 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.803104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.803130 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:17:07.803426 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.803439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.805202 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.805207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.806718 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.806734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.808309 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.808313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.808510 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.808562 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.808640 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.808759 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":157,"threads_started":1}
I20260812 11:17:07.809501 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.797369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.27.109.2:43455)
0812 11:17:07.801970 (+ 4601us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.801994 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.802024 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:17:07.802120 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.802126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.802443 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.802706 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:17:07.803643 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.805027 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.807083 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.808110 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.808936 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.809136 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.809221 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.809336 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4432,"thread_start_us":174,"threads_started":1}
I20260812 11:17:07.810307 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.794266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.27.109.1:38419)
0812 11:17:07.797857 (+ 3591us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.797867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.797903 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:17:07.798011 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.798017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.798311 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.798505 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:17:07.799317 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.800924 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.804845 (+ 3921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.806007 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.806151 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.807889 (+ 1738us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.809917 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.810103 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3417,"thread_start_us":121,"threads_started":1}
I20260812 11:17:07.808852 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.794122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:38419 (local address 127.0.0.1:53646)
0812 11:17:07.796755 (+ 2633us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.797184 (+ 429us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.797385 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.798739 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.798746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.798776 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:17:07.799115 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.799126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.801124 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.801130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.804612 (+ 3482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.804632 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.807990 (+ 3358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.807995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.808185 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.808245 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.808392 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.808609 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2213,"thread_start_us":227,"threads_started":1}
I20260812 11:17:07.813412 13685 tablet_service.cc:1467] Processing CreateTablet for tablet 8e024180d5c94b62a64802a8a6adad52 (DEFAULT_TABLE table=test-table [id=01e56aa8aa33483a9329acd87cbf74a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:17:07.816489 13685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e024180d5c94b62a64802a8a6adad52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:17:07.817166 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 8e024180d5c94b62a64802a8a6adad52 (DEFAULT_TABLE table=test-table [id=01e56aa8aa33483a9329acd87cbf74a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:17:07.817991 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.795951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:32823 (local address 127.0.0.1:35602)
0812 11:17:07.797335 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.797352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.797495 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.801233 (+ 3738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.801238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.801282 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:17:07.801438 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.801448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.814622 (+ 13174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.814628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.815872 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.815891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.817454 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.817459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.817571 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.817608 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.817703 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.817802 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"thread_start_us":180,"threads_started":1}
I20260812 11:17:07.818521 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e024180d5c94b62a64802a8a6adad52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:17:07.818847 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.796300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.27.109.3:32823)
0812 11:17:07.796809 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.796829 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.799384 (+ 2555us) server_negotiation.cc:407] Connection header received
0812 11:17:07.799501 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.799517 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.799804 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.800002 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:17:07.810871 (+ 10869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.814404 (+ 3533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.816137 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.817286 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.817490 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.817724 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.818403 (+ 679us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.818625 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":176,"threads_started":1}
I20260812 11:17:07.823305 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 8e024180d5c94b62a64802a8a6adad52 (DEFAULT_TABLE table=test-table [id=01e56aa8aa33483a9329acd87cbf74a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:17:07.825073 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e024180d5c94b62a64802a8a6adad52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:17:07.828984 14062 tablet_bootstrap.cc:492] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca: Bootstrap starting.
I20260812 11:17:07.832062 14062 tablet_bootstrap.cc:654] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca: Neither blocks nor log segments found. Creating new log.
I20260812 11:17:07.832677 14063 tablet_bootstrap.cc:492] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Bootstrap starting.
I20260812 11:17:07.833894 14062 log.cc:824] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca: Log is configured to *not* fsync() on all Append() calls
I20260812 11:17:07.839051 14063 tablet_bootstrap.cc:654] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Neither blocks nor log segments found. Creating new log.
I20260812 11:17:07.839426 14065 tablet_bootstrap.cc:492] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Bootstrap starting.
I20260812 11:17:07.841100 14063 log.cc:824] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:17:07.843570 14065 tablet_bootstrap.cc:654] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Neither blocks nor log segments found. Creating new log.
I20260812 11:17:07.844105 14062 tablet_bootstrap.cc:492] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca: No bootstrap required, opened a new log
I20260812 11:17:07.844375 14062 ts_tablet_manager.cc:1397] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260812 11:17:07.844762 14065 log.cc:824] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:17:07.849968 14062 raft_consensus.cc:348] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.850400 14062 raft_consensus.cc:374] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:17:07.850490 14062 raft_consensus.cc:729] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d207bc5103d48aeb46f653f4619d0ca, State: Initialized, Role: FOLLOWER
I20260812 11:17:07.851195 14062 consensus_queue.cc:260] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.852540 14062 ts_tablet_manager.cc:1428] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:17:07.853120 13567 heartbeater.cc:503] Master 127.27.109.62:45637 was elected leader, sending a full tablet report...
I20260812 11:17:07.854179 14065 tablet_bootstrap.cc:492] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: No bootstrap required, opened a new log
I20260812 11:17:07.854537 14065 ts_tablet_manager.cc:1397] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260812 11:17:07.855494 14063 tablet_bootstrap.cc:492] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: No bootstrap required, opened a new log
I20260812 11:17:07.855789 14063 ts_tablet_manager.cc:1397] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260812 11:17:07.860401 14063 raft_consensus.cc:348] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.860833 14063 raft_consensus.cc:374] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:17:07.860929 14063 raft_consensus.cc:729] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b7bb664e5f4a41a3402fa1f9b7437a, State: Initialized, Role: FOLLOWER
I20260812 11:17:07.861580 14063 consensus_queue.cc:260] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.861708 14065 raft_consensus.cc:348] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.862388 14065 raft_consensus.cc:374] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:17:07.862506 14065 raft_consensus.cc:729] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acfca20a8784973a3494c14309558f5, State: Initialized, Role: FOLLOWER
I20260812 11:17:07.863255 14065 consensus_queue.cc:260] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.863343 14063 ts_tablet_manager.cc:1428] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 11:17:07.863775 13804 heartbeater.cc:503] Master 127.27.109.62:45637 was elected leader, sending a full tablet report...
I20260812 11:17:07.864856 14065 ts_tablet_manager.cc:1428] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260812 11:17:07.865378 14034 heartbeater.cc:503] Master 127.27.109.62:45637 was elected leader, sending a full tablet report...
I20260812 11:17:07.898216 14077 raft_consensus.cc:482] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:17:07.898568 14077 raft_consensus.cc:504] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.900553 14077 leader_election.cc:288] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d207bc5103d48aeb46f653f4619d0ca (127.27.109.1:38419), 3acfca20a8784973a3494c14309558f5 (127.27.109.3:32823)
I20260812 11:17:07.908443 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.901940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:58509 (local address 127.27.109.3:32823)
0812 11:17:07.902113 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.902120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.902539 (+ 419us) server_negotiation.cc:407] Connection header received
0812 11:17:07.902766 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.902769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.902866 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.902927 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:17:07.903435 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.904289 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.906548 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.907319 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.907447 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.907526 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.908137 (+ 611us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.908248 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:17:07.908443 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.901090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:51487 (local address 127.27.109.1:38419)
0812 11:17:07.901463 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.901470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.901926 (+ 456us) server_negotiation.cc:407] Connection header received
0812 11:17:07.902247 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.902253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.902321 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.902390 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:17:07.903075 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.904073 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.906550 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.907319 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.907447 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.907526 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.908130 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.908248 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260812 11:17:07.909075 14075 raft_consensus.cc:482] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:17:07.909325 14075 raft_consensus.cc:504] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.909595 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.901417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:32823 (local address 127.27.109.2:58509)
0812 11:17:07.902429 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.902449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.902621 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.903122 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.903125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.903142 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:17:07.903265 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.903271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.904484 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.904488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.906352 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.906361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.907658 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.907661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.907914 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.907983 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.909159 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.909335 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":179,"thread_start_us":129,"threads_started":1}
I20260812 11:17:07.910601 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e024180d5c94b62a64802a8a6adad52" candidate_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acfca20a8784973a3494c14309558f5" is_pre_election: true
I20260812 11:17:07.910995 13930 raft_consensus.cc:2454] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b7bb664e5f4a41a3402fa1f9b7437a in term 0.
I20260812 11:17:07.911154 14075 leader_election.cc:288] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3acfca20a8784973a3494c14309558f5 (127.27.109.3:32823), d3b7bb664e5f4a41a3402fa1f9b7437a (127.27.109.2:43455)
I20260812 11:17:07.912298 13617 leader_election.cc:302] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acfca20a8784973a3494c14309558f5, d3b7bb664e5f4a41a3402fa1f9b7437a; no voters:
I20260812 11:17:07.912560 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.900894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:38419 (local address 127.27.109.2:51487)
0812 11:17:07.901794 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.901817 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.902060 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.902674 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.902677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.902697 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:17:07.902915 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.902925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.904247 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.904251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.906340 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.906365 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.907653 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.907657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.907912 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.907984 (+ 72us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.912270 (+ 4286us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.912385 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":159,"thread_start_us":234,"threads_started":1}
I20260812 11:17:07.912693 14077 raft_consensus.cc:2781] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:17:07.912801 14077 raft_consensus.cc:482] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:17:07.912856 14077 raft_consensus.cc:3037] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:17:07.913280 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e024180d5c94b62a64802a8a6adad52" candidate_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d207bc5103d48aeb46f653f4619d0ca" is_pre_election: true
I20260812 11:17:07.913511 13484 raft_consensus.cc:2454] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b7bb664e5f4a41a3402fa1f9b7437a in term 0.
I20260812 11:17:07.914608 14077 raft_consensus.cc:504] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.915109 14077 leader_election.cc:288] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [CANDIDATE]: Term 1 election: Requested vote from peers 5d207bc5103d48aeb46f653f4619d0ca (127.27.109.1:38419), 3acfca20a8784973a3494c14309558f5 (127.27.109.3:32823)
I20260812 11:17:07.916014 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e024180d5c94b62a64802a8a6adad52" candidate_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d207bc5103d48aeb46f653f4619d0ca"
I20260812 11:17:07.916172 13484 raft_consensus.cc:3037] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:17:07.917966 13484 raft_consensus.cc:2454] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b7bb664e5f4a41a3402fa1f9b7437a in term 1.
I20260812 11:17:07.918560 13619 leader_election.cc:302] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d207bc5103d48aeb46f653f4619d0ca, d3b7bb664e5f4a41a3402fa1f9b7437a; no voters:
I20260812 11:17:07.919364 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e024180d5c94b62a64802a8a6adad52" candidate_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acfca20a8784973a3494c14309558f5"
I20260812 11:17:07.919570 13930 raft_consensus.cc:3037] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:17:07.920802 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.912099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:56289 (local address 127.27.109.2:43455)
0812 11:17:07.912240 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.912246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.915900 (+ 3654us) server_negotiation.cc:407] Connection header received
0812 11:17:07.916141 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.916145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.916216 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.916272 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:17:07.916833 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.917609 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.919319 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.919954 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.920067 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.920136 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.920530 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.920635 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:17:07.921386 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.911560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43455 (local address 127.27.109.1:56289)
0812 11:17:07.915792 (+ 4232us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.915810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.916002 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.916456 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.916461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.916482 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:17:07.916691 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.916700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.917848 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.917853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.919173 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.919184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.920269 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.920272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.920374 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.920399 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.921058 (+ 659us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.921168 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4150,"thread_start_us":227,"threads_started":1}
I20260812 11:17:07.921519 13930 raft_consensus.cc:2454] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b7bb664e5f4a41a3402fa1f9b7437a in term 1.
I20260812 11:17:07.922278 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e024180d5c94b62a64802a8a6adad52" candidate_uuid: "5d207bc5103d48aeb46f653f4619d0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" is_pre_election: true
I20260812 11:17:07.922698 13717 raft_consensus.cc:2379] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d207bc5103d48aeb46f653f4619d0ca in current term 1: Already voted for candidate d3b7bb664e5f4a41a3402fa1f9b7437a in this term.
I20260812 11:17:07.924328 14077 raft_consensus.cc:2781] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:17:07.925091 14077 raft_consensus.cc:686] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 1 LEADER]: Becoming Leader. State: Replica: d3b7bb664e5f4a41a3402fa1f9b7437a, State: Running, Role: LEADER
I20260812 11:17:07.925671 14077 consensus_queue.cc:237] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } }
I20260812 11:17:07.927137 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.911718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:36599 (local address 127.27.109.3:32823)
0812 11:17:07.915527 (+ 3809us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.915534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.922079 (+ 6545us) server_negotiation.cc:407] Connection header received
0812 11:17:07.922317 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.922321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.922381 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.922432 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:17:07.922910 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.923668 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.925607 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.926228 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.926350 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.926427 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.926814 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.926951 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3700}
I20260812 11:17:07.927758 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.911560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:32823 (local address 127.27.109.1:36599)
0812 11:17:07.921991 (+ 10431us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:07.922010 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:17:07.922175 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:17:07.922601 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:17:07.922604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:17:07.922622 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:17:07.922729 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.922735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.923836 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.923840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.925397 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:17:07.925413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.926555 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.926561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:17:07.926658 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.926684 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:17:07.927342 (+ 658us) client_negotiation.cc:240] Negotiation successful
0812 11:17:07.927462 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10352,"mutex_wait_us":3489,"thread_start_us":167,"threads_started":1}
I20260812 11:17:07.928776 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e024180d5c94b62a64802a8a6adad52" candidate_uuid: "5d207bc5103d48aeb46f653f4619d0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acfca20a8784973a3494c14309558f5" is_pre_election: true
I20260812 11:17:07.929181 13930 raft_consensus.cc:2379] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d207bc5103d48aeb46f653f4619d0ca in current term 1: Already voted for candidate d3b7bb664e5f4a41a3402fa1f9b7437a in this term.
I20260812 11:17:07.929981 13379 leader_election.cc:302] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d207bc5103d48aeb46f653f4619d0ca; no voters: 3acfca20a8784973a3494c14309558f5, d3b7bb664e5f4a41a3402fa1f9b7437a
I20260812 11:17:07.930886 14075 raft_consensus.cc:2726] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 11:17:07.932327 13216 catalog_manager.cc:5495] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a reported cstate change: term changed from 0 to 1, leader changed from <none> to d3b7bb664e5f4a41a3402fa1f9b7437a (127.27.109.2). New cstate: current_term: 1 leader_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3b7bb664e5f4a41a3402fa1f9b7437a" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43455 } health_report { overall_health: HEALTHY } } }
W20260812 11:17:07.940341 13574 tablet.cc:2369] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:17:07.964790 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:07.960233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.27.109.2:43455)
0812 11:17:07.960619 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0812 11:17:07.960623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:17:07.960636 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:17:07.960741 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:17:07.960743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:17:07.960811 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:17:07.960947 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:17:07.961236 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.961797 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.962742 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:17:07.963071 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:17:07.963198 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:17:07.963583 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:17:07.964291 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:17:07.964443 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 11:17:07.964509 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 11:17:07.964586 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260812 11:17:07.975268 13930 raft_consensus.cc:1261] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 1 FOLLOWER]: Refusing update from remote peer d3b7bb664e5f4a41a3402fa1f9b7437a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:17:07.975417 13484 raft_consensus.cc:1261] T 8e024180d5c94b62a64802a8a6adad52 P 5d207bc5103d48aeb46f653f4619d0ca [term 1 FOLLOWER]: Refusing update from remote peer d3b7bb664e5f4a41a3402fa1f9b7437a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:17:07.976292 14077 consensus_queue.cc:1035] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3acfca20a8784973a3494c14309558f5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:17:07.976994 14077 consensus_queue.cc:1035] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d207bc5103d48aeb46f653f4619d0ca" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 38419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:17:07.988209 14036 tablet.cc:2369] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:17:07.988340 14093 mvcc.cc:205] Tried to move back new op lower bound from 7317640920970448896 to 7317640920790859776. Current Snapshot: MvccSnapshot[applied={T|T < 7317640920970448896}]
I20260812 11:17:07.998155 14095 mvcc.cc:205] Tried to move back new op lower bound from 7317640920970448896 to 7317640920790859776. Current Snapshot: MvccSnapshot[applied={T|T < 7317640920970448896}]
I20260812 11:17:08.011698 13216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:17:08.011961 13216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:17:08.016072 13216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27184c740f83474bbcf4a4603183f0e2: Sending DeleteTablet for 3 replicas of tablet 8e024180d5c94b62a64802a8a6adad52
I20260812 11:17:08.017978 13685 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e024180d5c94b62a64802a8a6adad52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:17:08 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:48574
I20260812 11:17:08.018396 13910 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e024180d5c94b62a64802a8a6adad52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:17:08 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:35602
I20260812 11:17:08.018595 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e024180d5c94b62a64802a8a6adad52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:17:08 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:53646
I20260812 11:17:08.020709 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 13347
I20260812 11:17:08.042958 14112 tablet_replica.cc:330] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: stopping tablet replica
I20260812 11:17:08.043604 14112 raft_consensus.cc:2229] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:17:08.044658 14112 raft_consensus.cc:2258] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:17:08.052605 14112 ts_tablet_manager.cc:1905] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:17:08.055778 14113 tablet_replica.cc:330] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: stopping tablet replica
I20260812 11:17:08.056437 14113 raft_consensus.cc:2229] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:17:08.056488 14112 ts_tablet_manager.cc:1918] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:17:08.056631 14112 log.cc:1196] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/8e024180d5c94b62a64802a8a6adad52
I20260812 11:17:08.057054 14112 ts_tablet_manager.cc:1939] T 8e024180d5c94b62a64802a8a6adad52 P d3b7bb664e5f4a41a3402fa1f9b7437a: Deleting consensus metadata
I20260812 11:17:08.057042 14113 raft_consensus.cc:2258] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:17:08.058775 13194 catalog_manager.cc:4841] TS d3b7bb664e5f4a41a3402fa1f9b7437a (127.27.109.2:43455): tablet 8e024180d5c94b62a64802a8a6adad52 (table test-table [id=01e56aa8aa33483a9329acd87cbf74a1]) successfully deleted
I20260812 11:17:08.059746 14113 ts_tablet_manager.cc:1905] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:17:08.064848 14113 ts_tablet_manager.cc:1918] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:17:08.065043 14113 log.cc:1196] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/8e024180d5c94b62a64802a8a6adad52
I20260812 11:17:08.066094 14113 ts_tablet_manager.cc:1939] T 8e024180d5c94b62a64802a8a6adad52 P 3acfca20a8784973a3494c14309558f5: Deleting consensus metadata
I20260812 11:17:08.068673 13190 catalog_manager.cc:4841] TS 3acfca20a8784973a3494c14309558f5 (127.27.109.3:32823): tablet 8e024180d5c94b62a64802a8a6adad52 (table test-table [id=01e56aa8aa33483a9329acd87cbf74a1]) successfully deleted
W20260812 11:17:08.081682 13195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:38419 (error 108)
I20260812 11:17:08.083329 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 13587
I20260812 11:17:08.106655 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:17:08.106143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:38419 (local address 127.0.0.1:53648)
0812 11:17:08.106431 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 11:17:08.106533 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:38419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260812 11:17:08.107467 13195 catalog_manager.cc:4584] TS 5d207bc5103d48aeb46f653f4619d0ca (127.27.109.1:38419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e024180d5c94b62a64802a8a6adad52: Network error: Client connection negotiation failed: client connection to 127.27.109.1:38419: connect: Connection refused (error 111)
Aug 12 11:17:08 dist-test-slave-7z32 krb5kdc[12481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533426, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:17:08.116400 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 13814
I20260812 11:17:08.155198 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 13164
I20260812 11:17:08.195160 28084 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 11:17:08.552390 28084 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 11:17:08.552565 28084 mini_ranger.cc:66] Stopping Ranger...
I20260812 11:17:08.910456 28084 mini_ranger.cc:68] Stopped Ranger
2026-08-12 11:17:08.910 UTC [12530] LOG: received smart shutdown request
2026-08-12 11:17:08.916 UTC [12530] LOG: background worker "logical replication launcher" (PID 12546) exited with exit code 1
2026-08-12 11:17:08.917 UTC [12535] LOG: shutting down
2026-08-12 11:17:09.162 UTC [12530] LOG: database system is shut down
2026-08-12T11:17:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (61015 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:17:09 dist-test-slave-7z32 krb5kdc[14137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:17:09 dist-test-slave-7z32 krb5kdc[14137](info): set up 2 sockets
Aug 12 11:17:09 dist-test-slave-7z32 krb5kdc[14137](info): commencing operation
krb5kdc: starting...
W20260812 11:17:10.756150 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.422s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:17:10 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:17:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:17:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
I20260812 11:17:11.012814 28084 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskK9g2P8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/postgres -l logfile start
2026-08-12 11:17:14.263 UTC [14189] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 11:17:14.264 UTC [14189] LOG: listening on IPv4 address "127.27.109.20", port 47487
2026-08-12 11:17:14.268 UTC [14189] LOG: listening on Unix socket "/tmp/.s.PGSQL.47487"
2026-08-12 11:17:14.295 UTC [14195] LOG: database system was shut down at 2026-08-12 11:17:12 UTC
2026-08-12 11:17:14.300 UTC [14189] LOG: database system is ready to accept connections
I20260812 11:17:15.572551 28084 mini_postgres.cc:96] Postgres bound to 47487
2026-08-12 11:17:15.582 UTC [14209] FATAL: database "slave" does not exist
127.27.109.20:47487 - accepting connections
I20260812 11:17:15.583427 28084 mini_ranger.cc:161] Starting Ranger...
I20260812 11:17:15.606621 28084 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 11:17:16.420600 28084 mini_ranger.cc:87] Created ranger Postgres database
I20260812 11:17:16.420750 28084 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-admin
2026-08-12 11:17:16,744 [I] DB FLAVOR :POSTGRES
2026-08-12 11:17:16,745 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:17:16,745 [I] Checking connection..
2026-08-12 11:17:16,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 11:17:17,073 [I] Checking connection passed.
2026-08-12 11:17:17,073 [I] --------- Verifying version history table ---------
2026-08-12 11:17:17,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:17:17,431 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:17:17,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:17:17,730 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:17:17,730 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 11:17:17,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 11:17:18.034 UTC [14326] WARNING: there is no transaction in progress
2026-08-12 11:17:18,049 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 11:17:18,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:17:18,360 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 11:17:18,360 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:17:18,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 11:17:18,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 11:17:18,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7z32.c.gcp-upstream.internal','N') ;"
2026-08-12 11:17:19,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:17:19,631 [I] Table x_portal_user does not exist in database ranger
2026-08-12 11:17:19,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:17:19,970 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 11:17:19,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:17:20,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:17:20,670 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 11:17:20,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 11:17:22.136 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.164 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.193 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.221 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.245 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.275 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.361 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.371 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.395 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.412 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.429 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.455 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.470 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.481 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.493 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:22.504 UTC [14512] WARNING: there is no transaction in progress
2026-08-12 11:17:23,526 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 11:17:23,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 11:17:23,833 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 11:17:23,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:17:24,142 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 11:17:24,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:17:24,457 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 11:17:24,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:17:24,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:17:25,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7z32.c.gcp-upstream.internal';"
2026-08-12 11:17:25,398 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 11:17:25,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 11:17:25,723 [I] DB_PATCHES have already been applied
I20260812 11:17:25.729916 28084 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-admin:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskK9g2P8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 11:17:25.730039 28084 mini_ranger.cc:193] Using host: 127.27.109.20
I20260812 11:17:25.737228 28084 mini_ranger.cc:227] Ranger admin URL: http://127.27.109.20:34293
Aug 12, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 11:17:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 11:17:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 11:17:28.996138 28084 mini_ranger.cc:160] Time spent starting Ranger: real 13.413s user 0.005s sys 0.018s
Aug 12, 2026 11:17:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 11:17:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 11:17:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 11:17:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 11:17:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 11:17:56.914793 28084 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 11:17:57 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533477, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 11:17:57.048318 28084 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 11:17:57.065541 28084 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 11:17:58.168079 28084 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 11:17:58.168200 28084 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms
I20260812 11:17:58.168226 28084 mini_ranger_kms.cc:227] Using postgres at 127.27.109.20:47487
2026-08-12 11:17:58,391 [I] DB FLAVOR :POSTGRES
2026-08-12 11:17:58,391 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:17:58,391 [I] Checking connection
2026-08-12 11:17:58,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 11:17:58,759 [I] connection success
2026-08-12 11:17:58,759 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 11:17:58,759 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 11:17:58,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 11:17:59,165 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 11:17:59,166 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:17:59,166 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 11:17:59,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:47487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 11:17:59,607 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 11:18:00.363371 28084 mini_ranger_kms.cc:310] Created kms service
I20260812 11:18:00.656672 28084 mini_ranger_kms.cc:326] Created kudu user
I20260812 11:18:00.728514 28084 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 11:18:00 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533477, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:01.284242 28084 mini_ranger_kms.cc:384] Added ranger policy
I20260812 11:18:01.284482 28084 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms:/tmp/dist-test-taskK9g2P8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskK9g2P8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 11:18:01.284505 28084 mini_ranger_kms.cc:240] Using host: 127.27.109.20
I20260812 11:18:01.292076 28084 mini_ranger_kms.cc:276] Ranger KMS PID: 14828
I20260812 11:18:01.292250 28084 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.109.20:34479
Aug 12, 2026 11:18:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:18:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:18:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 11:18:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 11:18:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 11:18:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 11:18:04 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533484, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:04.376855 28084 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.329s user 0.000s sys 0.211s
I20260812 11:18:04.377044 28084 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 11:18:04.377167 28084 mini_ranger_kms.cc:401] 127.27.109.20:34479/kms/v1/keys
Aug 12 11:18:04 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533484, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 12 11:18:05 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533484, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 12 11:18:08 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:18:08.500208 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35627
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:35627
--ranger_config_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:08.630312 14910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:08.630646 14910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:08.630689 14910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:08.635010 14910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:08.635102 14910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:08.635123 14910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:08.635138 14910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:08.635154 14910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:08.641219 14910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.109.20:34479/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:35627
--ranger_config_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:08.642567 14910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:08.643774 14910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:08.655277 14918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:08.655668 14910 server_base.cc:1034] running on GCE node
W20260812 11:18:08.655272 14916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:08.655305 14922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:08.656838 14910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:08.658594 14910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:08.662150 14910 hybrid_clock.cc:648] HybridClock initialized: now 1786533488662025 us; error 85 us; skew 500 ppm
Aug 12 11:18:08 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533488, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:08.672641 14910 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:08.674291 14910 webserver.cc:460] Webserver started at http://127.27.109.62:33191/ using document root <none> and password file <none>
I20260812 11:18:08.675523 14910 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:08.675665 14910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:08.675997 14910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:18:08 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533488, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:08.757474 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "0a09f881611a4072abfd4b04565759a9"
format_stamp: "Formatted at 2026-08-12 11:18:08 on dist-test-slave-7z32"
server_key: "a1905f6b6c4a2da167a206188536bfea"
server_key_iv: "8733ee6165435e82a315e56eb86fcd9a"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:08.759224 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "0a09f881611a4072abfd4b04565759a9"
format_stamp: "Formatted at 2026-08-12 11:18:08 on dist-test-slave-7z32"
server_key: "a1905f6b6c4a2da167a206188536bfea"
server_key_iv: "8733ee6165435e82a315e56eb86fcd9a"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:08.766935 14910 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 11:18:08.794915 14931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:08.797801 14910 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:18:08.798030 14910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0a09f881611a4072abfd4b04565759a9"
format_stamp: "Formatted at 2026-08-12 11:18:08 on dist-test-slave-7z32"
server_key: "a1905f6b6c4a2da167a206188536bfea"
server_key_iv: "8733ee6165435e82a315e56eb86fcd9a"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:08.798280 14910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:08.819821 14910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:08.824186 14910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:08.824556 14910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:08.834370 14910 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:35627
I20260812 11:18:08.834472 15016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:35627 every 8 connection(s)
I20260812 11:18:08.835770 14910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:08.841063 15019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:08.845713 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 14910
I20260812 11:18:08.845854 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:18:08.846215 28084 external_mini_cluster.cc:1428] Setting key 8bba75414660078b4d882c32af1c95c0
I20260812 11:18:08.851410 15019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Bootstrap starting.
I20260812 11:18:08.855392 15019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:08.856755 15019 log.cc:824] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:18:08 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:08.860025 15019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: No bootstrap required, opened a new log
I20260812 11:18:08.864641 15019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:08.865000 15019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:08.865061 15019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a09f881611a4072abfd4b04565759a9, State: Initialized, Role: FOLLOWER
I20260812 11:18:08.865654 15019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:08.865844 15019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:18:08.865906 15019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:18:08.866012 15019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:08.866441 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:08.848269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.27.109.62:35627)
0812 11:18:08.848804 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:08.848822 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:08.848869 (+ 47us) server_negotiation.cc:407] Connection header received
0812 11:18:08.849746 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:08.849769 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:08.850190 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:08.850551 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:08.852001 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:08.853195 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:08.854708 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:08.855461 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:08.855702 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:08.860080 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:08.860097 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:08.860123 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:08.860164 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:08.862512 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:08.863243 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:08.863249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:08.863257 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:08.863431 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:08.863763 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:08.863766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:08.863768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:08.864388 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:08.864619 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:08.865033 (+ 414us) server_negotiation.cc:299] Negotiation successful
0812 11:18:08.865384 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":195,"threads_started":1}
I20260812 11:18:08.867440 15019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:08.868113 15019 leader_election.cc:302] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a09f881611a4072abfd4b04565759a9; no voters:
I20260812 11:18:08.868613 15019 leader_election.cc:288] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:18:08.868873 15024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:08.869223 15024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0a09f881611a4072abfd4b04565759a9, State: Running, Role: LEADER
I20260812 11:18:08.869832 15024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:08.870491 15019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:08.872658 15028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a09f881611a4072abfd4b04565759a9. Latest consensus state: current_term: 1 leader_uuid: "0a09f881611a4072abfd4b04565759a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } } }
I20260812 11:18:08.872938 15028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:08.872658 15026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a09f881611a4072abfd4b04565759a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } } }
I20260812 11:18:08.873409 15026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:08.873509 15019 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:18:08.874106 15049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:08.883200 15049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:08.891819 15049 catalog_manager.cc:1347] Generated new cluster ID: 14889e45700b44149d0434a3370c7cbf
I20260812 11:18:08.891958 15049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:18:08.934927 15049 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:18:08.936607 15049 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:18:08.947021 15049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Generated new TSK 0
I20260812 11:18:08.948094 15049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:18:09.812230 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35627
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:10.001622 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:10.002876 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:10.003078 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:10.010166 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:10.010497 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:10.011153 15091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:18:10.020737 15091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.109.20:34479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:10.022892 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:10.024721 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:10.044214 15109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:10.045279 15113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:10.045728 15108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:10.049321 15091 server_base.cc:1034] running on GCE node
I20260812 11:18:10.050254 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:10.051836 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:10.059360 15091 hybrid_clock.cc:648] HybridClock initialized: now 1786533490059209 us; error 172 us; skew 500 ppm
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:10.065552 15091 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:18:10.067597 15091 webserver.cc:460] Webserver started at http://127.27.109.1:36495/ using document root <none> and password file <none>
I20260812 11:18:10.068768 15091 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:10.068998 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:10.069365 15091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:10.097250 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "40383793143a4883a59badc9bd552e72"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "f561dae9e53bfc3c2433acb2b7da7f51"
server_key_iv: "1b0d153adba3b314f792cd8bdbe5b88b"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:10.098336 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "40383793143a4883a59badc9bd552e72"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "f561dae9e53bfc3c2433acb2b7da7f51"
server_key_iv: "1b0d153adba3b314f792cd8bdbe5b88b"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:10.106225 15091 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 11:18:10.138535 15134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:10.140822 15091 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260812 11:18:10.141010 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "40383793143a4883a59badc9bd552e72"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "f561dae9e53bfc3c2433acb2b7da7f51"
server_key_iv: "1b0d153adba3b314f792cd8bdbe5b88b"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:10.141183 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:10.160362 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:10.164060 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:10.164305 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:10.165119 15091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:10.166446 15091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:10.166553 15091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:10.166663 15091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:10.166699 15091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:10.181824 15091 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:33653
I20260812 11:18:10.181838 15335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:33653 every 8 connection(s)
I20260812 11:18:10.184695 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:18:10.190749 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 15091
I20260812 11:18:10.190958 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:18:10.191308 28084 external_mini_cluster.cc:1428] Setting key df4bf0c3cf11d6160e1986989df0557b
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:10.218325 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:10.191709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:45159 (local address 127.27.109.62:35627)
0812 11:18:10.194927 (+ 3218us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:10.194934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:10.194963 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:18:10.195029 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:10.195033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:10.195106 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:10.195231 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:10.197859 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.199009 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.201657 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.202131 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.202272 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:10.207830 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:10.207838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:10.207841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:10.207890 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:10.210973 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:10.215089 (+ 4116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:10.215098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:10.215100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:10.215414 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:10.216224 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:10.216229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:10.216231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:10.216573 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:10.216675 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:10.217771 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0812 11:18:10.218025 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3042,"thread_start_us":237,"threads_started":1}
I20260812 11:18:10.221354 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:10.189840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35627 (local address 127.27.109.1:45159)
0812 11:18:10.192481 (+ 2641us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:10.192602 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:10.194107 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:10.195752 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:10.195767 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:10.196356 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:10.197475 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:10.197511 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.199295 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.199299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:10.201337 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:10.201365 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.202653 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.202660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:10.203049 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:10.204399 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:10.204433 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:10.207395 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:10.211419 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:10.211438 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:10.211476 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:10.212086 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:10.215649 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:10.215654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:10.215656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:10.216047 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:10.216790 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:10.216802 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:10.217362 (+ 560us) client_negotiation.cc:769] Sending connection context
0812 11:18:10.219158 (+ 1796us) client_negotiation.cc:240] Negotiation successful
0812 11:18:10.219787 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":279,"threads_started":1}
I20260812 11:18:10.224272 15336 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35627
I20260812 11:18:10.224661 15336 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:10.225721 15336 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:10.228271 14949 ts_manager.cc:194] Registered new tserver with Master: 40383793143a4883a59badc9bd552e72 (127.27.109.1:33653)
I20260812 11:18:10.230154 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:45159
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:18:10.260735 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35627
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:10.452805 15344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:10.453542 15344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:10.453715 15344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:10.461014 15344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:10.461298 15344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:10.461511 15344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:18:10.471099 15344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.109.20:34479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:10.473577 15344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:10.475505 15344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:10.492322 15355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:10.492136 15358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:10.492838 15354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:10.493418 15344 server_base.cc:1034] running on GCE node
I20260812 11:18:10.494318 15344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:10.495733 15344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:10.499205 15344 hybrid_clock.cc:648] HybridClock initialized: now 1786533490499067 us; error 133 us; skew 500 ppm
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:10.505044 15344 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:18:10.507462 15344 webserver.cc:460] Webserver started at http://127.27.109.2:46477/ using document root <none> and password file <none>
I20260812 11:18:10.508471 15344 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:10.508618 15344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:10.508927 15344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:10.533582 15344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "8531f763dbc94d81acc58c1e80af75a4"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "b74e5f200d50b74d59b5d8b7590106c7"
server_key_iv: "12eaba526779c0933bcf3294a590f7c6"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:10.534981 15344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "8531f763dbc94d81acc58c1e80af75a4"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "b74e5f200d50b74d59b5d8b7590106c7"
server_key_iv: "12eaba526779c0933bcf3294a590f7c6"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:10.544298 15344 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260812 11:18:10.579887 15372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:10.582582 15344 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.001s
I20260812 11:18:10.582870 15344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "8531f763dbc94d81acc58c1e80af75a4"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "b74e5f200d50b74d59b5d8b7590106c7"
server_key_iv: "12eaba526779c0933bcf3294a590f7c6"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:10.583133 15344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:10.607750 15344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:10.613865 15344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:10.614223 15344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:10.615437 15344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:10.617040 15344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:10.617136 15344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:10.617218 15344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:10.617256 15344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:10.635634 15344 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43169
I20260812 11:18:10.636984 15571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43169 every 8 connection(s)
I20260812 11:18:10.637535 15344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:18:10.643602 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 15344
I20260812 11:18:10.643769 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:18:10.644090 28084 external_mini_cluster.cc:1428] Setting key 9d64750a277a9d67739ff29d732b2ced
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:10.679756 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:10.644811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:35717 (local address 127.27.109.62:35627)
0812 11:18:10.645331 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:10.645338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:10.653082 (+ 7744us) server_negotiation.cc:407] Connection header received
0812 11:18:10.655563 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:10.655570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:10.655710 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:10.655868 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:10.657943 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.658986 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.664854 (+ 5868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.665222 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.665358 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:10.673722 (+ 8364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:10.673727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:10.673731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:10.673778 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:10.676167 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:10.677135 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:10.677140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:10.677142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:10.677252 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:10.677782 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:10.677786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:10.677787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:10.678086 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:10.678186 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:10.679342 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0812 11:18:10.679559 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260812 11:18:10.682704 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:10.642729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35627 (local address 127.27.109.2:35717)
0812 11:18:10.652713 (+ 9984us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:10.652831 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:10.654733 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:10.656297 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:10.656311 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:10.656818 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:10.657670 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:10.657696 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.661332 (+ 3636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.661341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:10.664519 (+ 3178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:10.664566 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:10.667193 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:10.667202 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:10.667699 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:10.669438 (+ 1739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:10.669474 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:10.673196 (+ 3722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:10.676425 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:10.676437 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:10.676467 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:10.676977 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:10.677397 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:10.677400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:10.677402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:10.677648 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:10.678300 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:10.678310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:10.678955 (+ 645us) client_negotiation.cc:769] Sending connection context
0812 11:18:10.680370 (+ 1415us) client_negotiation.cc:240] Negotiation successful
0812 11:18:10.680861 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9590,"thread_start_us":354,"threads_started":1}
I20260812 11:18:10.684852 15572 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35627
I20260812 11:18:10.685304 15572 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:10.686120 15572 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:10.687912 14949 ts_manager.cc:194] Registered new tserver with Master: 8531f763dbc94d81acc58c1e80af75a4 (127.27.109.2:43169)
I20260812 11:18:10.688853 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:35717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:18:10.707188 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35627
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
W20260812 11:18:10.935935 15583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:10.936455 15583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:10.936529 15583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:10.943830 15583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:10.943964 15583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:10.944053 15583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:18:10.951751 15583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.109.20:34479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:10.953161 15583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:10.954387 15583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:10.967597 15590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:10.967607 15588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:10.968109 15592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:10.968129 15583 server_base.cc:1034] running on GCE node
I20260812 11:18:10.968989 15583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:10.969889 15583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:10.973321 15583 hybrid_clock.cc:648] HybridClock initialized: now 1786533490973252 us; error 89 us; skew 500 ppm
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:10.986969 15583 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:18:10.988413 15583 webserver.cc:460] Webserver started at http://127.27.109.3:37241/ using document root <none> and password file <none>
I20260812 11:18:10.989231 15583 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:10.989383 15583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:10.989657 15583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:18:10 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:11.012560 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "30a53b540b4d4a8b898910742bfbf65c"
format_stamp: "Formatted at 2026-08-12 11:18:11 on dist-test-slave-7z32"
server_key: "ec384cae1a4505a441fdcb03e4f62b9a"
server_key_iv: "c425945380f8e2fd3b66e509f7ade0f8"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:11.013620 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "30a53b540b4d4a8b898910742bfbf65c"
format_stamp: "Formatted at 2026-08-12 11:18:11 on dist-test-slave-7z32"
server_key: "ec384cae1a4505a441fdcb03e4f62b9a"
server_key_iv: "c425945380f8e2fd3b66e509f7ade0f8"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:11.022223 15583 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260812 11:18:11.058008 15601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:11.060035 15583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 11:18:11.060812 15583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "30a53b540b4d4a8b898910742bfbf65c"
format_stamp: "Formatted at 2026-08-12 11:18:11 on dist-test-slave-7z32"
server_key: "ec384cae1a4505a441fdcb03e4f62b9a"
server_key_iv: "c425945380f8e2fd3b66e509f7ade0f8"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:11.061440 15583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:11.090318 15583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:11.094792 15583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:11.095132 15583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:11.095988 15583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:11.097086 15583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:11.097164 15583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:11.097235 15583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:11.097290 15583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:11.112085 15583 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:39353
I20260812 11:18:11.113343 15583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:18:11.117750 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:39353 every 8 connection(s)
I20260812 11:18:11.121622 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 15583
I20260812 11:18:11.121806 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:18:11.122212 28084 external_mini_cluster.cc:1428] Setting key c6126684306f2f8e6bd7e129cedc01b0
Aug 12 11:18:11 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:11.138767 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:11.115926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39071 (local address 127.27.109.62:35627)
0812 11:18:11.116091 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:11.116097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:11.117279 (+ 1182us) server_negotiation.cc:407] Connection header received
0812 11:18:11.119356 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:11.119362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:11.119470 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:11.119554 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:11.123050 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:11.124056 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:11.126724 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:11.127238 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:11.127381 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:11.131942 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:11.131948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:11.131952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:11.131996 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:11.134431 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:11.135697 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:11.135703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:11.135706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:11.135865 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:11.136492 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:11.136496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:11.136498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:11.136824 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:11.136933 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:11.138288 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0812 11:18:11.138540 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:18:11.142133 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:11.116473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35627 (local address 127.27.109.3:39071)
0812 11:18:11.117082 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:11.117146 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:11.118862 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:11.120950 (+ 2088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:11.120969 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:11.121622 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:11.122711 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:11.122746 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:11.124348 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:11.124352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:11.126474 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:11.126496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:11.127724 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:11.127730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:11.128020 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:11.129020 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:11.129050 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:11.131587 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:11.134719 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:11.134732 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:11.134764 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:11.135504 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:11.136023 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:11.136027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:11.136030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:11.136320 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:11.137027 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:11.137039 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:11.137796 (+ 757us) client_negotiation.cc:769] Sending connection context
0812 11:18:11.139561 (+ 1765us) client_negotiation.cc:240] Negotiation successful
0812 11:18:11.140207 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":187,"threads_started":1}
I20260812 11:18:11.145149 15791 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35627
I20260812 11:18:11.145560 15791 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:11.146487 15791 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:11.149936 14949 ts_manager.cc:194] Registered new tserver with Master: 30a53b540b4d4a8b898910742bfbf65c (127.27.109.3:39353)
I20260812 11:18:11.151109 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:39071
I20260812 11:18:11.160719 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:18:11.161628 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 15091
I20260812 11:18:11.239231 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 15344
I20260812 11:18:11.301811 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 15583
I20260812 11:18:11.334751 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 14910
I20260812 11:18:11.367550 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35627
--webserver_interface=127.27.109.62
--webserver_port=33191
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:35627
--ranger_config_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:11.536516 15798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:11.537045 15798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:11.537106 15798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:11.543212 15798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:11.543349 15798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:11.543372 15798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:11.543393 15798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:11.543412 15798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:11.551921 15798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:35627
--ranger_config_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=33191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:11.553614 15798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:11.555094 15798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:18:11.567251 15798 server_base.cc:1034] running on GCE node
W20260812 11:18:11.567122 15810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:11.567145 15819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:11.567195 15814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:11.568439 15798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:11.570103 15798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:11.571452 15798 hybrid_clock.cc:648] HybridClock initialized: now 1786533491571419 us; error 62 us; skew 500 ppm
Aug 12 11:18:11 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:11.575846 15798 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:11.577335 15798 webserver.cc:460] Webserver started at http://127.27.109.62:33191/ using document root <none> and password file <none>
I20260812 11:18:11.578190 15798 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:11.578308 15798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:18:11 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:11.663805 15798 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:11.667074 15838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:11.668869 15798 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:18:11.669113 15798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0a09f881611a4072abfd4b04565759a9"
format_stamp: "Formatted at 2026-08-12 11:18:08 on dist-test-slave-7z32"
server_key: "a1905f6b6c4a2da167a206188536bfea"
server_key_iv: "8733ee6165435e82a315e56eb86fcd9a"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:11.669756 15798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:11.689320 15798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:11.693360 15798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:11.693940 15798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:11.703506 15798 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:35627
I20260812 11:18:11.703506 15925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:35627 every 8 connection(s)
I20260812 11:18:11.705103 15798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:11.713197 15926 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:18:11.713851 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 15798
I20260812 11:18:11.714581 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:33653
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:35627
--webserver_port=36495
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:11.717403 15926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Bootstrap starting.
I20260812 11:18:11.744653 15926 log.cc:824] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:11.754937 15926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:18:11.755587 15926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Bootstrap complete.
I20260812 11:18:11.761425 15926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:11.762238 15926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a09f881611a4072abfd4b04565759a9, State: Initialized, Role: FOLLOWER
I20260812 11:18:11.763095 15926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:11.763257 15926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:18:11.763366 15926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:18:11.763499 15926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:18:11.766611 15926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:11.767272 15926 leader_election.cc:302] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a09f881611a4072abfd4b04565759a9; no voters:
I20260812 11:18:11.767778 15926 leader_election.cc:288] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:18:11.768061 15933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:18:11.768457 15933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [term 2 LEADER]: Becoming Leader. State: Replica: 0a09f881611a4072abfd4b04565759a9, State: Running, Role: LEADER
I20260812 11:18:11.768898 15933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } }
I20260812 11:18:11.769629 15926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:11.770551 15933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a09f881611a4072abfd4b04565759a9. Latest consensus state: current_term: 2 leader_uuid: "0a09f881611a4072abfd4b04565759a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } } }
I20260812 11:18:11.770757 15933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:11.771327 15934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a09f881611a4072abfd4b04565759a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09f881611a4072abfd4b04565759a9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35627 } } }
I20260812 11:18:11.771334 15941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:11.771593 15934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:11.777097 15941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:11.777988 15926 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:18:11.778009 15941 catalog_manager.cc:1259] Loaded cluster ID: 14889e45700b44149d0434a3370c7cbf
I20260812 11:18:11.778460 15941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:18:11.781587 15941 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:18:11.782553 15941 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Loaded TSK: 0
I20260812 11:18:11.784298 15941 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:18:11.907496 15930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:11.907857 15930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:11.907940 15930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:11.912838 15930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:11.912981 15930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:11.913087 15930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:18:11.922278 15930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:33653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=36495
--tserver_master_addrs=127.27.109.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:11.924060 15930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:11.927289 15930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:11.939998 16002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:11.940088 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:11.940331 15996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:11.941742 15930 server_base.cc:1034] running on GCE node
I20260812 11:18:11.942557 15930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:11.943521 15930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:11.944777 15930 hybrid_clock.cc:648] HybridClock initialized: now 1786533491944715 us; error 65 us; skew 500 ppm
Aug 12 11:18:11 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:11.949460 15930 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:18:11.950922 15930 webserver.cc:460] Webserver started at http://127.27.109.1:36495/ using document root <none> and password file <none>
I20260812 11:18:11.951655 15930 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:11.951812 15930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:18:11 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:11.988330 15930 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:12.002454 16014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.005942 15930 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.001s sys 0.000s
I20260812 11:18:12.006224 15930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "40383793143a4883a59badc9bd552e72"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "f561dae9e53bfc3c2433acb2b7da7f51"
server_key_iv: "1b0d153adba3b314f792cd8bdbe5b88b"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:12.007344 15930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:12.032146 15930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:12.038277 15930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:12.038609 15930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:12.039822 15930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:12.041517 15930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:12.041612 15930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.041687 15930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:12.041724 15930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.059875 15930 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:33653
I20260812 11:18:12.061903 15930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:18:12.063694 16211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:33653 every 8 connection(s)
I20260812 11:18:12.065886 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 15930
I20260812 11:18:12.066486 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:43169
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:35627
--webserver_port=46477
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:18:12 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:12.101244 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:12.067665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35627 (local address 127.27.109.1:36699)
0812 11:18:12.069271 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:12.069379 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:12.073079 (+ 3700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:12.079872 (+ 6793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:12.079892 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:12.080489 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:12.081663 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:12.081715 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.085080 (+ 3365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.085086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:12.086863 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:12.086883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.088543 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.088547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:12.088779 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:12.089784 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:12.089812 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:12.091926 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:12.095603 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:12.095613 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:12.095636 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:12.096396 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:12.097016 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:12.097020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:12.097022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:12.097267 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:12.098460 (+ 1193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:12.098469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:12.098948 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 11:18:12.099206 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 11:18:12.099742 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"thread_start_us":268,"threads_started":1}
I20260812 11:18:12.105711 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:12.070295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:36699 (local address 127.27.109.62:35627)
0812 11:18:12.071106 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:12.071136 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:12.071216 (+ 80us) server_negotiation.cc:407] Connection header received
0812 11:18:12.074287 (+ 3071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:12.074308 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:12.074908 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:12.082356 (+ 7448us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:12.082949 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.084789 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.087111 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.087978 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.088183 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:12.092304 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:12.092314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:12.092339 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:12.092389 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:12.095328 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.096630 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.096640 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.096648 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.096869 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.097411 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.097418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.097421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.098293 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:12.102428 (+ 4135us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:12.102737 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 11:18:12.103283 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":188,"threads_started":1}
I20260812 11:18:12.110512 16212 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35627
I20260812 11:18:12.111055 16212 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:12.112246 16212 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
W20260812 11:18:12.116182 16212 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:35627 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 11:18:12.116441 16212 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:12.116748 16212 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:12.118170 16212 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:12.118448 16212 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
W20260812 11:18:12.119447 16212 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:18:12.268930 16216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:12.269452 16216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:12.269575 16216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:12.276572 16216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:12.276736 16216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:12.276841 16216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:18:12.285200 16216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:43169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=46477
--tserver_master_addrs=127.27.109.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:12.287042 16216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:12.288542 16216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:12.310510 16227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:12.310503 16230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:12.312479 16216 server_base.cc:1034] running on GCE node
W20260812 11:18:12.312832 16226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:12.313470 16216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:12.314489 16216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:12.315876 16216 hybrid_clock.cc:648] HybridClock initialized: now 1786533492315804 us; error 98 us; skew 500 ppm
Aug 12 11:18:12 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:12.320869 16216 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:18:12.322909 16216 webserver.cc:460] Webserver started at http://127.27.109.2:46477/ using document root <none> and password file <none>
I20260812 11:18:12.324096 16216 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:12.324285 16216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:18:12 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:12.350797 16216 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 11:18:12.355291 16259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.357789 16216 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260812 11:18:12.358129 16216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "8531f763dbc94d81acc58c1e80af75a4"
format_stamp: "Formatted at 2026-08-12 11:18:10 on dist-test-slave-7z32"
server_key: "b74e5f200d50b74d59b5d8b7590106c7"
server_key_iv: "12eaba526779c0933bcf3294a590f7c6"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:12.359519 16216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:12.385032 16216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:12.390352 16216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:12.390689 16216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:12.391805 16216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:12.393416 16216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:12.393504 16216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.393580 16216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:12.393625 16216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.422278 16216 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43169
I20260812 11:18:12.422358 16459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43169 every 8 connection(s)
I20260812 11:18:12.424878 16216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:18:12.435184 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 16216
I20260812 11:18:12.435882 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:34479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:39353
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:35627
--webserver_port=37241
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:18:12 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:12.454267 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:12.428622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:50821 (local address 127.27.109.62:35627)
0812 11:18:12.428916 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:12.428926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:12.430236 (+ 1310us) server_negotiation.cc:407] Connection header received
0812 11:18:12.432088 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:12.432093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:12.432182 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:12.432337 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:12.436632 (+ 4295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.437543 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.439857 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.440237 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.440334 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:12.448805 (+ 8471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:12.448810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:12.448815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:12.448857 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:12.451012 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.451901 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.451905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.451907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.452019 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.452543 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.452548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.452549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.452909 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:12.453016 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:12.453909 (+ 893us) server_negotiation.cc:299] Negotiation successful
0812 11:18:12.454079 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 11:18:12.456238 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:12.429158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35627 (local address 127.27.109.2:50821)
0812 11:18:12.429938 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:12.430021 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:12.431653 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:12.432819 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:12.432834 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:12.433848 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:12.436286 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:12.436344 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.437824 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.437828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:12.439618 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:12.439639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.440605 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.440609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:12.440876 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:12.441712 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:12.441738 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:12.448455 (+ 6717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:12.451204 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:12.451213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:12.451238 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:12.451727 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:12.452154 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:12.452158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:12.452160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:12.452405 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:12.453103 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:12.453111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:12.453542 (+ 431us) client_negotiation.cc:769] Sending connection context
0812 11:18:12.454519 (+ 977us) client_negotiation.cc:240] Negotiation successful
0812 11:18:12.454991 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":251,"threads_started":1}
I20260812 11:18:12.459240 16460 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35627
I20260812 11:18:12.459722 16460 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:12.460747 16460 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
W20260812 11:18:12.463642 16460 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:35627 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 11:18:12.463846 16460 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:12.464169 16460 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:12.465670 16460 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:12.465914 16460 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
W20260812 11:18:12.467136 16460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:18:12.645190 16466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:12.645645 16466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:12.645732 16466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:12.652292 16466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:12.652407 16466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:12.652509 16466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:18:12.661217 16466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:34479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:39353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=37241
--tserver_master_addrs=127.27.109.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:12.662979 16466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:12.664644 16466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:12.680234 16476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:12.680310 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:12.680737 16488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:12.682912 16466 server_base.cc:1034] running on GCE node
I20260812 11:18:12.683776 16466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:12.684774 16466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:12.688097 16466 hybrid_clock.cc:648] HybridClock initialized: now 1786533492688035 us; error 74 us; skew 500 ppm
Aug 12 11:18:12 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:12.693011 16466 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:18:12.694651 16466 webserver.cc:460] Webserver started at http://127.27.109.3:37241/ using document root <none> and password file <none>
I20260812 11:18:12.695747 16466 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:12.695986 16466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:18:12 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:12.733784 16466 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:18:12.738256 16498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.740530 16466 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:18:12.740756 16466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "30a53b540b4d4a8b898910742bfbf65c"
format_stamp: "Formatted at 2026-08-12 11:18:11 on dist-test-slave-7z32"
server_key: "ec384cae1a4505a441fdcb03e4f62b9a"
server_key_iv: "c425945380f8e2fd3b66e509f7ade0f8"
server_key_version: "kuduclusterkey@0"
I20260812 11:18:12.741703 16466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:12.762396 16466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:12.767086 16466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:12.767341 16466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:12.768204 16466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:12.769562 16466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:12.769647 16466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.769714 16466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:12.769743 16466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:12.786415 16466 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:39353
I20260812 11:18:12.787177 16723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:39353 every 8 connection(s)
I20260812 11:18:12.788062 16466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:18:12.793186 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 16466
Aug 12 11:18:12 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:12.807806 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:12.794336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.27.109.62:35627)
0812 11:18:12.794891 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:12.794906 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:12.795029 (+ 123us) server_negotiation.cc:407] Connection header received
0812 11:18:12.795201 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:12.795205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:12.795264 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:12.795397 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:12.796466 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.797366 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.798799 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.799283 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.799440 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:12.802118 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:12.802123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:12.802126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:12.802200 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:12.804896 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.805700 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.805704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.805706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.805818 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.806214 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.806220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.806227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.806546 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:12.806869 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:12.807359 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 11:18:12.807535 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":424,"thread_start_us":176,"threads_started":1}
I20260812 11:18:12.809193 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:12.790857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:46533 (local address 127.27.109.62:35627)
0812 11:18:12.791046 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:12.791050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:12.792436 (+ 1386us) server_negotiation.cc:407] Connection header received
0812 11:18:12.793919 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:12.793931 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:12.794046 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:12.794199 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:12.796357 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.797356 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.799341 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.799751 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.799915 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:12.803744 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:12.803749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:12.803752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:12.803822 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:12.805876 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.806875 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.806881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.806884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.806999 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:12.807540 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:12.807546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:12.807551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:12.807847 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:12.807944 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:12.808859 (+ 915us) server_negotiation.cc:299] Negotiation successful
0812 11:18:12.809016 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":67}
I20260812 11:18:12.810397 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:12.791367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35627 (local address 127.27.109.3:46533)
0812 11:18:12.792156 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:12.792239 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:12.793492 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:12.794737 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:12.794751 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:12.795355 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:12.796162 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:12.796183 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.797666 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.797673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:12.799124 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:12.799141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:12.800013 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:12.800020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:12.800345 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:12.801095 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:12.801126 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:12.803425 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:12.806105 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:12.806117 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:12.806144 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:12.806679 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:12.807144 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:12.807147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:12.807150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:12.807397 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:12.808028 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:12.808036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:12.808490 (+ 454us) client_negotiation.cc:769] Sending connection context
0812 11:18:12.808748 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 11:18:12.809180 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":243,"threads_started":1}
I20260812 11:18:12.812142 16725 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35627
I20260812 11:18:12.812738 16725 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:12.813802 16725 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:12.816190 15866 ts_manager.cc:194] Registered new tserver with Master: 30a53b540b4d4a8b898910742bfbf65c (127.27.109.3:39353)
I20260812 11:18:12.818224 15866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:46533
I20260812 11:18:13.120035 16212 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:13.120510 16212 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:13.121793 15866 ts_manager.cc:194] Registered new tserver with Master: 40383793143a4883a59badc9bd552e72 (127.27.109.1:33653)
I20260812 11:18:13.122802 15866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:36699
Aug 12 11:18:13 dist-test-slave-7z32 krb5kdc[14137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:13.467481 16460 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:13.468019 16460 heartbeater.cc:511] Master 127.27.109.62:35627 requested a full tablet report, sending...
I20260812 11:18:13.469348 15866 ts_manager.cc:194] Registered new tserver with Master: 8531f763dbc94d81acc58c1e80af75a4 (127.27.109.2:43169)
I20260812 11:18:13.470515 15866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:50821
I20260812 11:18:13.471890 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:18:13.484659 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.474000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.27.109.62:35627)
0812 11:18:13.474374 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:13.474380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:13.474442 (+ 62us) server_negotiation.cc:407] Connection header received
0812 11:18:13.474627 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:13.474630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:13.474698 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:13.474853 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:13.475711 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.476572 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.477659 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.478009 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.478130 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.479932 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:13.479937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:13.479941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:13.479979 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:13.482132 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:13.482696 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:13.482700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:13.482702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:13.482815 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:13.483179 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:13.483182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:13.483186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:13.483493 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:13.483661 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:13.483993 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 11:18:13.484430 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20260812 11:18:13.489902 15866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:18:13.494272 15866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:18:13.521015 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.508824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43169 (local address 127.0.0.1:40026)
0812 11:18:13.510513 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:13.510586 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:13.510750 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:13.512084 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:13.512087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:13.512104 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:13.512200 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.512207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.513779 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.513783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.517126 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.517143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.520219 (+ 3076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.520224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.520392 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.520449 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 11:18:13.520598 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 11:18:13.520790 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1585,"mutex_wait_us":1973,"spinlock_wait_cycles":29312,"thread_start_us":216,"threads_started":1}
I20260812 11:18:13.521991 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.508765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39353 (local address 127.0.0.1:41476)
0812 11:18:13.510513 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:13.510586 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:13.510750 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:13.511709 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:13.511716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:13.511739 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:13.511960 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.511969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.513780 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.513783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.515600 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.515619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.521490 (+ 5871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.521495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.521622 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.521651 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:18:13.521717 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 11:18:13.521817 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"mutex_wait_us":3,"spinlock_wait_cycles":1280,"thread_start_us":170,"threads_started":1}
I20260812 11:18:13.522969 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.509946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.27.109.3:39353)
0812 11:18:13.511140 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:13.511149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:13.511176 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:18:13.511270 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:13.511276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:13.511522 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:13.512122 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:13.512279 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.513598 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.517386 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.519656 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.522334 (+ 2678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.522574 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:13.522641 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 11:18:13.522776 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":167,"threads_started":1}
I20260812 11:18:13.523272 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.508932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33653 (local address 127.0.0.1:50442)
0812 11:18:13.511051 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:13.511065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:13.511169 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:13.512617 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:13.512620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:13.512644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:13.512814 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.512822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.516501 (+ 3679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.516505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.517865 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.517886 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.522605 (+ 4719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.522611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.522780 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.522847 (+ 67us) client_negotiation.cc:769] Sending connection context
0812 11:18:13.522926 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:18:13.523057 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2056,"thread_start_us":102,"threads_started":1}
I20260812 11:18:13.523573 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.509970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.27.109.2:43169)
0812 11:18:13.511477 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:13.511484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:13.511509 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:18:13.511587 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:13.511595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:13.511766 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:13.511920 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:13.512350 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.513598 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.517813 (+ 4215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.518805 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.519908 (+ 1103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.520124 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:13.523190 (+ 3066us) server_negotiation.cc:299] Negotiation successful
0812 11:18:13.523324 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":136,"threads_started":1}
I20260812 11:18:13.527344 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.509982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.27.109.1:33653)
0812 11:18:13.511858 (+ 1876us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:13.511870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:13.511913 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:18:13.512066 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:13.512076 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:13.512401 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:13.513016 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:13.513204 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.516284 (+ 3080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.519689 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.522078 (+ 2389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.526550 (+ 4472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.526888 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:13.526988 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 11:18:13.527145 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677,"thread_start_us":165,"threads_started":1}
I20260812 11:18:13.528908 16323 tablet_service.cc:1467] Processing CreateTablet for tablet adde3a5b749648d3b07497c2c488f3fd (DEFAULT_TABLE table=test-table [id=5137cf7df3174f89b66bcb54e304e0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:13.530592 16323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde3a5b749648d3b07497c2c488f3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:13.530375 16600 tablet_service.cc:1467] Processing CreateTablet for tablet adde3a5b749648d3b07497c2c488f3fd (DEFAULT_TABLE table=test-table [id=5137cf7df3174f89b66bcb54e304e0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:13.531894 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde3a5b749648d3b07497c2c488f3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:13.532992 16087 tablet_service.cc:1467] Processing CreateTablet for tablet adde3a5b749648d3b07497c2c488f3fd (DEFAULT_TABLE table=test-table [id=5137cf7df3174f89b66bcb54e304e0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:13.534528 16087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde3a5b749648d3b07497c2c488f3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:13.547286 16770 tablet_bootstrap.cc:492] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: Bootstrap starting.
I20260812 11:18:13.549795 16771 tablet_bootstrap.cc:492] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72: Bootstrap starting.
I20260812 11:18:13.553817 16770 tablet_bootstrap.cc:654] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:13.553817 16771 tablet_bootstrap.cc:654] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:13.555207 16770 log.cc:824] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:13.555207 16771 log.cc:824] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:13.556457 16773 tablet_bootstrap.cc:492] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Bootstrap starting.
I20260812 11:18:13.558605 16770 tablet_bootstrap.cc:492] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: No bootstrap required, opened a new log
I20260812 11:18:13.558945 16770 ts_tablet_manager.cc:1397] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260812 11:18:13.561637 16773 tablet_bootstrap.cc:654] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:13.563043 16773 log.cc:824] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:13.566607 16771 tablet_bootstrap.cc:492] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72: No bootstrap required, opened a new log
I20260812 11:18:13.566607 16773 tablet_bootstrap.cc:492] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: No bootstrap required, opened a new log
I20260812 11:18:13.566977 16771 ts_tablet_manager.cc:1397] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260812 11:18:13.566977 16773 ts_tablet_manager.cc:1397] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260812 11:18:13.572888 16771 raft_consensus.cc:348] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.573374 16771 raft_consensus.cc:374] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:13.573452 16771 raft_consensus.cc:729] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40383793143a4883a59badc9bd552e72, State: Initialized, Role: FOLLOWER
I20260812 11:18:13.574127 16770 raft_consensus.cc:348] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.574522 16770 raft_consensus.cc:374] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:13.574591 16770 raft_consensus.cc:729] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8531f763dbc94d81acc58c1e80af75a4, State: Initialized, Role: FOLLOWER
I20260812 11:18:13.575335 16771 consensus_queue.cc:260] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.575728 16773 raft_consensus.cc:348] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.576061 16773 raft_consensus.cc:374] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:13.576125 16773 raft_consensus.cc:729] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a53b540b4d4a8b898910742bfbf65c, State: Initialized, Role: FOLLOWER
I20260812 11:18:13.576664 16773 consensus_queue.cc:260] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.577001 16771 ts_tablet_manager.cc:1428] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260812 11:18:13.577638 16770 consensus_queue.cc:260] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.577786 16773 ts_tablet_manager.cc:1428] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260812 11:18:13.578208 16725 heartbeater.cc:503] Master 127.27.109.62:35627 was elected leader, sending a full tablet report...
I20260812 11:18:13.578863 16770 ts_tablet_manager.cc:1428] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: Time spent starting tablet: real 0.020s user 0.001s sys 0.006s
I20260812 11:18:13.579281 16212 heartbeater.cc:503] Master 127.27.109.62:35627 was elected leader, sending a full tablet report...
I20260812 11:18:13.581956 16460 heartbeater.cc:503] Master 127.27.109.62:35627 was elected leader, sending a full tablet report...
W20260812 11:18:13.677911 16461 tablet.cc:2369] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:13.707497 16785 raft_consensus.cc:482] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:13.707890 16785 raft_consensus.cc:504] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
Aug 12 11:18:13 dist-test-slave-7z32 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260812 11:18:13.713514 16785 leader_election.cc:288] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a53b540b4d4a8b898910742bfbf65c (127.27.109.3:39353), 40383793143a4883a59badc9bd552e72 (127.27.109.1:33653)
I20260812 11:18:13.727916 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.712201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:45447 (local address 127.27.109.3:39353)
0812 11:18:13.712354 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:13.712359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:13.715204 (+ 2845us) server_negotiation.cc:407] Connection header received
0812 11:18:13.715840 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:13.715845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:13.715992 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:13.716067 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:13.716830 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.717928 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.726006 (+ 8078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.726753 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.727017 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.727086 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:13.727610 (+ 524us) server_negotiation.cc:299] Negotiation successful
0812 11:18:13.727735 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:18:13.728616 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.711257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39353 (local address 127.27.109.2:45447)
0812 11:18:13.715060 (+ 3803us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:13.715081 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:13.715374 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:13.716290 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:13.716293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:13.716318 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:13.716576 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.716585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.721458 (+ 4873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.721462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.725664 (+ 4202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.725693 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.727317 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.727321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.727440 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.727467 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:18:13.728339 (+ 872us) client_negotiation.cc:240] Negotiation successful
0812 11:18:13.728433 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3673,"thread_start_us":266,"threads_started":1}
I20260812 11:18:13.730379 16641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde3a5b749648d3b07497c2c488f3fd" candidate_uuid: "8531f763dbc94d81acc58c1e80af75a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a53b540b4d4a8b898910742bfbf65c" is_pre_election: true
I20260812 11:18:13.731015 16641 raft_consensus.cc:2454] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8531f763dbc94d81acc58c1e80af75a4 in term 0.
I20260812 11:18:13.732019 16261 leader_election.cc:302] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a53b540b4d4a8b898910742bfbf65c, 8531f763dbc94d81acc58c1e80af75a4; no voters:
I20260812 11:18:13.732573 16785 raft_consensus.cc:2781] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:18:13.732689 16785 raft_consensus.cc:482] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:18:13.732750 16785 raft_consensus.cc:3037] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:13.733706 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.713336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:55877 (local address 127.27.109.1:33653)
0812 11:18:13.713470 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:13.713475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:13.719258 (+ 5783us) server_negotiation.cc:407] Connection header received
0812 11:18:13.719528 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:13.719532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:13.719604 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:13.719658 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:13.726703 (+ 7045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.727523 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.731653 (+ 4130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.732279 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.732443 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.732507 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:13.733437 (+ 930us) server_negotiation.cc:299] Negotiation successful
0812 11:18:13.733550 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:18:13.734530 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.712813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33653 (local address 127.27.109.2:55877)
0812 11:18:13.719126 (+ 6313us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:13.719148 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:13.719375 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:13.719828 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:13.719831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:13.719849 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:13.719998 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.720005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.727761 (+ 7756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.727764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.731443 (+ 3679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:13.731468 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.733052 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.733057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:13.733181 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.733212 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:18:13.734141 (+ 929us) client_negotiation.cc:240] Negotiation successful
0812 11:18:13.734274 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6188,"thread_start_us":157,"threads_started":1}
I20260812 11:18:13.734710 16785 raft_consensus.cc:504] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.735399 16785 leader_election.cc:288] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [CANDIDATE]: Term 1 election: Requested vote from peers 30a53b540b4d4a8b898910742bfbf65c (127.27.109.3:39353), 40383793143a4883a59badc9bd552e72 (127.27.109.1:33653)
I20260812 11:18:13.735507 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde3a5b749648d3b07497c2c488f3fd" candidate_uuid: "8531f763dbc94d81acc58c1e80af75a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40383793143a4883a59badc9bd552e72" is_pre_election: true
I20260812 11:18:13.736023 16130 raft_consensus.cc:2454] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8531f763dbc94d81acc58c1e80af75a4 in term 0.
I20260812 11:18:13.736270 16112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde3a5b749648d3b07497c2c488f3fd" candidate_uuid: "8531f763dbc94d81acc58c1e80af75a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40383793143a4883a59badc9bd552e72"
I20260812 11:18:13.736486 16112 raft_consensus.cc:3037] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:13.737282 16641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde3a5b749648d3b07497c2c488f3fd" candidate_uuid: "8531f763dbc94d81acc58c1e80af75a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a53b540b4d4a8b898910742bfbf65c"
I20260812 11:18:13.737495 16641 raft_consensus.cc:3037] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:13.738464 16112 raft_consensus.cc:2454] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8531f763dbc94d81acc58c1e80af75a4 in term 1.
I20260812 11:18:13.739413 16641 raft_consensus.cc:2454] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8531f763dbc94d81acc58c1e80af75a4 in term 1.
I20260812 11:18:13.739418 16265 leader_election.cc:302] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40383793143a4883a59badc9bd552e72, 8531f763dbc94d81acc58c1e80af75a4; no voters:
I20260812 11:18:13.739979 16785 raft_consensus.cc:2781] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:13.740352 16785 raft_consensus.cc:686] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8531f763dbc94d81acc58c1e80af75a4, State: Running, Role: LEADER
I20260812 11:18:13.740947 16785 consensus_queue.cc:237] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } }
I20260812 11:18:13.746942 15866 catalog_manager.cc:5495] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8531f763dbc94d81acc58c1e80af75a4 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "8531f763dbc94d81acc58c1e80af75a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8531f763dbc94d81acc58c1e80af75a4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260812 11:18:13.777420 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.771967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.27.109.2:43169)
0812 11:18:13.772110 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:13.772134 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:13.772151 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:18:13.772268 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:13.772271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:13.772344 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:13.772481 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:18:13.772906 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.773745 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.775074 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:13.775425 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:13.775531 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:13.775999 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:18:13.776647 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:18:13.776778 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:13.777158 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 11:18:13.777226 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:18:13.786603 16112 raft_consensus.cc:1261] T adde3a5b749648d3b07497c2c488f3fd P 40383793143a4883a59badc9bd552e72 [term 1 FOLLOWER]: Refusing update from remote peer 8531f763dbc94d81acc58c1e80af75a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:13.786603 16641 raft_consensus.cc:1261] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 1 FOLLOWER]: Refusing update from remote peer 8531f763dbc94d81acc58c1e80af75a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:13.787600 16785 consensus_queue.cc:1035] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a53b540b4d4a8b898910742bfbf65c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:13.788100 16793 consensus_queue.cc:1035] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40383793143a4883a59badc9bd552e72" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:18:13.790020 16729 tablet.cc:2369] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:13.804087 16798 mvcc.cc:205] Tried to move back new op lower bound from 7317641190537469952 to 7317641190371536896. Current Snapshot: MvccSnapshot[applied={T|T < 7317641190537469952}]
I20260812 11:18:13.807682 16799 mvcc.cc:205] Tried to move back new op lower bound from 7317641190537469952 to 7317641190371536896. Current Snapshot: MvccSnapshot[applied={T|T < 7317641190537469952}]
I20260812 11:18:13.816926 16802 mvcc.cc:205] Tried to move back new op lower bound from 7317641190537469952 to 7317641190371536896. Current Snapshot: MvccSnapshot[applied={T|T < 7317641190537469952}]
I20260812 11:18:13.835803 15866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:13.836066 15866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:13.839409 15866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a09f881611a4072abfd4b04565759a9: Sending DeleteTablet for 3 replicas of tablet adde3a5b749648d3b07497c2c488f3fd
I20260812 11:18:13.841163 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet adde3a5b749648d3b07497c2c488f3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:13 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:41476
I20260812 11:18:13.842068 16087 tablet_service.cc:1514] Processing DeleteTablet for tablet adde3a5b749648d3b07497c2c488f3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:13 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:50442
I20260812 11:18:13.843309 16323 tablet_service.cc:1514] Processing DeleteTablet for tablet adde3a5b749648d3b07497c2c488f3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:13 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40026
I20260812 11:18:13.844216 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 15930
I20260812 11:18:13.859247 16816 tablet_replica.cc:330] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: stopping tablet replica
I20260812 11:18:13.860121 16816 raft_consensus.cc:2229] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:18:13.860806 16816 raft_consensus.cc:2258] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:13.863490 16814 tablet_replica.cc:330] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: stopping tablet replica
I20260812 11:18:13.868418 16814 raft_consensus.cc:2229] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:18:13.869207 16814 raft_consensus.cc:2258] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:13.874336 16816 ts_tablet_manager.cc:1905] T adde3a5b749648d3b07497c2c488f3fd P 8531f763dbc94d81acc58c1e80af75a4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 11:18:13.877532 15846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:33653 (error 108)
I20260812 11:18:13.879129 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 16216
I20260812 11:18:13.952057 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.951265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33653 (local address 127.0.0.1:50450)
0812 11:18:13.951697 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:13.951882 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260812 11:18:13.952463 15846 catalog_manager.cc:4584] TS 40383793143a4883a59badc9bd552e72 (127.27.109.1:33653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adde3a5b749648d3b07497c2c488f3fd: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33653: connect: Connection refused (error 111)
I20260812 11:18:13.964085 16814 ts_tablet_manager.cc:1905] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:13.969689 16814 ts_tablet_manager.cc:1918] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:13.969897 16814 log.cc:1196] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/adde3a5b749648d3b07497c2c488f3fd
I20260812 11:18:13.970975 16814 ts_tablet_manager.cc:1939] T adde3a5b749648d3b07497c2c488f3fd P 30a53b540b4d4a8b898910742bfbf65c: Deleting consensus metadata
I20260812 11:18:13.975310 15840 catalog_manager.cc:4841] TS 30a53b540b4d4a8b898910742bfbf65c (127.27.109.3:39353): tablet adde3a5b749648d3b07497c2c488f3fd (table test-table [id=5137cf7df3174f89b66bcb54e304e0f1]) successfully deleted
I20260812 11:18:13.993140 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 16466
I20260812 11:18:13.993726 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:13.993266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43169 (local address 127.0.0.1:40044)
0812 11:18:13.993533 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:13.993601 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.2:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260812 11:18:14.038309 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:14.035730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33653 (local address 127.0.0.1:50458)
0812 11:18:14.038034 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:14.038156 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2117}
I20260812 11:18:14.040009 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:14.039540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33653 (local address 127.0.0.1:50466)
0812 11:18:14.039727 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:14.039785 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 11:18:14.055549 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 15798
I20260812 11:18:14.082073 28084 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 11:18:14.442498 28084 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 11:18:14.442713 28084 mini_ranger.cc:66] Stopping Ranger...
I20260812 11:18:14.802469 28084 mini_ranger.cc:68] Stopped Ranger
2026-08-12 11:18:14.802 UTC [14189] LOG: received smart shutdown request
2026-08-12 11:18:14.812 UTC [14189] LOG: background worker "logical replication launcher" (PID 14205) exited with exit code 1
2026-08-12 11:18:14.812 UTC [14198] LOG: shutting down
2026-08-12 11:18:15.123 UTC [14189] LOG: database system is shut down
2026-08-12T11:18:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (65957 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:18:15 dist-test-slave-7z32 krb5kdc[16830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:18:15 dist-test-slave-7z32 krb5kdc[16830](info): set up 2 sockets
Aug 12 11:18:15 dist-test-slave-7z32 krb5kdc[16830](info): commencing operation
krb5kdc: starting...
W20260812 11:18:16.694881 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.403s user 0.011s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:18:16 dist-test-slave-7z32 krb5kdc[16830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:18:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:18:16.865404 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45737
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45737,127.27.109.61:43071,127.27.109.60:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:17.021605 16850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:17.022115 16850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:17.022174 16850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:17.027330 16850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:17.027442 16850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:17.027464 16850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:17.027484 16850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:17.027503 16850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:17.034904 16850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45737,127.27.109.61:43071,127.27.109.60:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:17.036661 16850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:17.038059 16850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:17.050060 16867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:17.050061 16862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:17.050308 16850 server_base.cc:1034] running on GCE node
W20260812 11:18:17.050082 16865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:17.051357 16850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:17.053032 16850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:17.054250 16850 hybrid_clock.cc:648] HybridClock initialized: now 1786533497054210 us; error 75 us; skew 500 ppm
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:17.058300 16850 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:17.059799 16850 webserver.cc:460] Webserver started at http://127.27.109.62:43635/ using document root <none> and password file <none>
I20260812 11:18:17.060631 16850 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:17.060745 16850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:17.061046 16850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:17.063915 16850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "1e859a681ccc4b5a8c023eb0e24c863b"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "f368a5c89feb14eb2ee653a5053e36b2"
server_key_iv: "92ccfaf16103f844578aff37ebf5f8a9"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.064674 16850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "1e859a681ccc4b5a8c023eb0e24c863b"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "f368a5c89feb14eb2ee653a5053e36b2"
server_key_iv: "92ccfaf16103f844578aff37ebf5f8a9"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.069972 16850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:18:17.073781 16879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:17.075827 16850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:17.076063 16850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "1e859a681ccc4b5a8c023eb0e24c863b"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "f368a5c89feb14eb2ee653a5053e36b2"
server_key_iv: "92ccfaf16103f844578aff37ebf5f8a9"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.076246 16850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:17.099540 16850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:17.104400 16850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:17.104768 16850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:17.115561 16850 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:45737
I20260812 11:18:17.115662 16974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:45737 every 8 connection(s)
I20260812 11:18:17.117096 16850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:17.117947 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 16850
I20260812 11:18:17.118288 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:18:17.118687 28084 external_mini_cluster.cc:1428] Setting key d9428fe2b5c13ec104cc798f2f141c98
I20260812 11:18:17.121713 16975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:17.126578 16975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.61@KRBTEST.COM" created.
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:17.149945 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.129629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.27.109.62:45737)
0812 11:18:17.130406 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.130430 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.130689 (+ 259us) server_negotiation.cc:407] Connection header received
0812 11:18:17.131987 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.131994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.132374 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.132733 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.133903 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.135555 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.137152 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.137986 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.138274 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.142236 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.142246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.142262 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.142299 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.144812 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.145829 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.145837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.145839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.146079 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.146612 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.146616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.146618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.147387 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.147554 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.148324 (+ 770us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.148850 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":555,"thread_start_us":316,"threads_started":1}
I20260812 11:18:17.149945 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.129629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45737 (local address 127.0.0.1:51672)
0812 11:18:17.130394 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.130463 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.131519 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.132689 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.132697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.133269 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.133707 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.133727 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.135886 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.135892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.136937 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.136957 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.138266 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.138271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.138433 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.139275 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.139296 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.141925 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.145039 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.145061 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.145080 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.145641 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.146228 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.146234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.146239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.146418 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.147628 (+ 1210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.147637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.147859 (+ 222us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.148121 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.148668 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":205,"threads_started":1}
I20260812 11:18:17.154115 16975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:18:17.155503 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.155066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41866)
0812 11:18:17.155264 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.155380 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 11:18:17.155964 16880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
I20260812 11:18:17.157490 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.157151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41876)
0812 11:18:17.157325 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.157389 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 11:18:17.157673 16975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43071: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
I20260812 11:18:17.160073 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:43071
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45737,127.27.109.61:43071,127.27.109.60:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:17.207011 16975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } attempt: 1
I20260812 11:18:17.208472 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.207767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41884)
0812 11:18:17.208036 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.208162 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 11:18:17.209676 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.209343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41886)
0812 11:18:17.209490 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.209578 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 11:18:17.209983 16975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43071: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
I20260812 11:18:17.278303 16975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } attempt: 2
I20260812 11:18:17.279724 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.279195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41888)
0812 11:18:17.279476 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.279590 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 11:18:17.281004 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.280611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41892)
0812 11:18:17.280778 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.280903 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 11:18:17.281239 16975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43071: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
W20260812 11:18:17.315071 16991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:17.315418 16991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:17.315464 16991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:17.319794 16991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:17.319895 16991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:17.319916 16991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:17.319934 16991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:17.319950 16991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:17.325722 16991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45737,127.27.109.61:43071,127.27.109.60:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:43071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:17.327109 16991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:17.328230 16991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:17.338503 17001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:17.338402 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:17.338536 17002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:17.339267 16991 server_base.cc:1034] running on GCE node
I20260812 11:18:17.340004 16991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:17.341426 16991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:17.342815 16991 hybrid_clock.cc:648] HybridClock initialized: now 1786533497342793 us; error 51 us; skew 500 ppm
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:17.346798 16991 init.cc:377] Logged in from keytab as kudu/127.27.109.61@KRBTEST.COM (short username kudu)
I20260812 11:18:17.348116 16991 webserver.cc:460] Webserver started at http://127.27.109.61:43883/ using document root <none> and password file <none>
I20260812 11:18:17.348884 16991 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:17.348996 16991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:17.349284 16991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:17.351867 16991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data/instance:
uuid: "e0542b67e3134936a2a74bfdb99ecbac"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "2aae10191181943219a4c1cd3f44e5b9"
server_key_iv: "bf48b40eb9eb27bb7dbf27e7f752a914"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.352559 16991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/wal/instance:
uuid: "e0542b67e3134936a2a74bfdb99ecbac"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "2aae10191181943219a4c1cd3f44e5b9"
server_key_iv: "bf48b40eb9eb27bb7dbf27e7f752a914"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.357640 16991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:18:17.361498 17015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:17.363606 16991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:18:17.363852 16991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/wal
uuid: "e0542b67e3134936a2a74bfdb99ecbac"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "2aae10191181943219a4c1cd3f44e5b9"
server_key_iv: "bf48b40eb9eb27bb7dbf27e7f752a914"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.364032 16991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:17.372426 16975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } attempt: 3
I20260812 11:18:17.373651 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.373212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41900)
0812 11:18:17.373440 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.373543 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 11:18:17.374686 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.374401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41912)
0812 11:18:17.374536 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.374610 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 11:18:17.374908 16975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43071: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43071: connect: Connection refused (error 111)
I20260812 11:18:17.380024 16991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:17.385011 16991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:17.385243 16991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:17.394757 16991 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:43071
I20260812 11:18:17.394874 17093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:43071 every 8 connection(s)
I20260812 11:18:17.396134 16991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
I20260812 11:18:17.400760 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:17.402794 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 16991
I20260812 11:18:17.403065 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-1/wal/instance
I20260812 11:18:17.403409 28084 external_mini_cluster.cc:1428] Setting key 00843a333babbe18338eebe7156ecf93
I20260812 11:18:17.406680 17094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.60@KRBTEST.COM" created.
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:17.427124 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.409872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.27.109.62:45737)
0812 11:18:17.410089 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.410095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.411406 (+ 1311us) server_negotiation.cc:407] Connection header received
0812 11:18:17.412804 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.412809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.412896 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.413031 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.414999 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.415597 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.417230 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.417544 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.417786 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.421998 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.422003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.422006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.422044 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.424184 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.425036 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.425039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.425041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.425130 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.425548 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.425552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.425554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.425818 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.425943 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.426691 (+ 748us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.426916 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 11:18:17.428395 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.410398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45737 (local address 127.0.0.1:51674)
0812 11:18:17.411177 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.411248 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.412514 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.413398 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.413407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.413959 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.414662 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.414681 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.415813 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.415818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.417054 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.417067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.417848 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.417853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.418073 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.418997 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.419035 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.421639 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.424382 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.424395 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.424425 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.424850 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.425246 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.425256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.425258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.425435 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.425944 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.425956 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.426361 (+ 405us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.426614 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.427130 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":250,"threads_started":1}
I20260812 11:18:17.431021 17094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:18:17.439597 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:38079
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45737,127.27.109.61:43071,127.27.109.60:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:17.443953 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.431925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41916)
0812 11:18:17.432136 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.432152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.432266 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.433100 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.433103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.433465 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.433626 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.433634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.434797 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.434800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.435802 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.435828 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.436662 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.436665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.436783 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.437535 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.437549 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.439378 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.442022 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.442028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.442031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.442259 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.442589 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.442592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.442594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.442677 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.443352 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.443354 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.443445 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.443569 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.443744 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 11:18:17.443975 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.432141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.27.109.61:43071)
0812 11:18:17.432546 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.432561 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.432587 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:18:17.432683 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.432692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.432918 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.433056 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.433813 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.434645 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.435971 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.436500 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.436605 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.439571 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.439586 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.439604 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.439628 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.441863 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.442389 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.442400 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.442402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.442482 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.442809 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.442812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.442813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.443229 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.443398 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.443675 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.443823 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260812 11:18:17.446271 17094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:18:17.447790 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.447240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59004)
0812 11:18:17.447465 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.447625 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 11:18:17.448176 17016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
I20260812 11:18:17.449414 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.449069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59018)
0812 11:18:17.449233 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.449313 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 11:18:17.449627 17094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:38079: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
I20260812 11:18:17.499024 17094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } attempt: 1
I20260812 11:18:17.500572 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.499935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59020)
0812 11:18:17.500257 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.500393 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 11:18:17.501833 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.501390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59030)
0812 11:18:17.501610 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.501712 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260812 11:18:17.502079 17094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:38079: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
I20260812 11:18:17.518123 16975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } attempt: 4
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:18:17.531404 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.519114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.27.109.61:43071)
0812 11:18:17.519299 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.519303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.519535 (+ 232us) server_negotiation.cc:407] Connection header received
0812 11:18:17.519739 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.519742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.519828 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.519949 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.520693 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.521406 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.523547 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.523867 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.523985 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.527033 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.527037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.527039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.527077 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.529068 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.529766 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.529770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.529771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.529867 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.530271 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.530275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.530276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.530503 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.530605 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.531065 (+ 460us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.531204 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:18:17.531396 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.518930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41922)
0812 11:18:17.519328 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.519354 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.519546 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.520009 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.520013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.520305 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.520501 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.520509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.521565 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.521569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.523351 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.523368 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.524036 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.524040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.524178 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.524904 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.524921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.526772 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.529241 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.529246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.529248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.529605 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.529988 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.529993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.529997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.530126 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.530622 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.530625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.530788 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.530948 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.531161 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260812 11:18:17.533293 16975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:18:17.534308 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.533880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59036)
0812 11:18:17.534098 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.534188 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 11:18:17.535244 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.534955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59040)
0812 11:18:17.535101 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.535165 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 11:18:17.535524 16975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:38079: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
I20260812 11:18:17.570310 17094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } attempt: 2
I20260812 11:18:17.571565 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.571166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59048)
0812 11:18:17.571376 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.571469 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 11:18:17.572477 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.572239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59050)
0812 11:18:17.572357 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.572418 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 11:18:17.572613 17094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:38079: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
I20260812 11:18:17.594803 16975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } attempt: 1
I20260812 11:18:17.596206 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.595748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59056)
0812 11:18:17.596007 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.596103 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260812 11:18:17.597735 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.597214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59072)
0812 11:18:17.597499 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.597602 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260812 11:18:17.598004 16975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:38079: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
W20260812 11:18:17.604100 17107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:17.604542 17107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:17.604604 17107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:17.609623 17107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:17.609755 17107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:17.609778 17107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:17.609805 17107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:17.609823 17107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:17.616799 17107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45737,127.27.109.61:43071,127.27.109.60:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:17.618515 17107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:17.620133 17107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:17.631969 17117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:17.632162 17120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:17.632313 17107 server_base.cc:1034] running on GCE node
W20260812 11:18:17.632588 17115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:17.633610 17107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:17.635361 17107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:17.636655 17107 hybrid_clock.cc:648] HybridClock initialized: now 1786533497636607 us; error 76 us; skew 500 ppm
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:17.640861 17107 init.cc:377] Logged in from keytab as kudu/127.27.109.60@KRBTEST.COM (short username kudu)
I20260812 11:18:17.642503 17107 webserver.cc:460] Webserver started at http://127.27.109.60:40913/ using document root <none> and password file <none>
I20260812 11:18:17.643512 17107 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:17.643663 17107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:17.644013 17107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:17.647509 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data/instance:
uuid: "49afe1c0ad0640828cb18080a8bc3463"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "adbd28ceb6351d20af55d6aa4358c618"
server_key_iv: "1a69ab8894104ebc33b1feb97dd825ab"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.648392 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/wal/instance:
uuid: "49afe1c0ad0640828cb18080a8bc3463"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "adbd28ceb6351d20af55d6aa4358c618"
server_key_iv: "1a69ab8894104ebc33b1feb97dd825ab"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.654460 17107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 11:18:17.658764 17132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:17.661055 17107 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:17.661338 17107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/wal
uuid: "49afe1c0ad0640828cb18080a8bc3463"
format_stamp: "Formatted at 2026-08-12 11:18:17 on dist-test-slave-7z32"
server_key: "adbd28ceb6351d20af55d6aa4358c618"
server_key_iv: "1a69ab8894104ebc33b1feb97dd825ab"
server_key_version: "encryptionkey@0"
I20260812 11:18:17.661530 17107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:17.663815 17094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } attempt: 3
I20260812 11:18:17.665127 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.664631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59088)
0812 11:18:17.664916 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.665015 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 11:18:17.665329 16975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } attempt: 2
I20260812 11:18:17.666290 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.665986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59106)
0812 11:18:17.666168 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.666224 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 11:18:17.666326 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.665878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59094)
0812 11:18:17.666094 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.666194 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 11:18:17.666525 17094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:38079: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
I20260812 11:18:17.667243 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.666984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59120)
0812 11:18:17.667105 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.667176 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:18:17.667392 16975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:38079: Network error: Client connection negotiation failed: client connection to 127.27.109.60:38079: connect: Connection refused (error 111)
I20260812 11:18:17.688257 17107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:17.692857 17107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:17.693102 17107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:17.703027 17107 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:38079
I20260812 11:18:17.703114 17225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:38079 every 8 connection(s)
I20260812 11:18:17.704427 17107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
I20260812 11:18:17.708457 17228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:17.713078 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 17107
I20260812 11:18:17.713275 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533282802791-28084-0/minicluster-data/master-2/wal/instance
I20260812 11:18:17.713596 28084 external_mini_cluster.cc:1428] Setting key 879702e49c1f370a857ffc806972ec32
I20260812 11:18:17.714115 17228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:17.730154 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.716707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.27.109.62:45737)
0812 11:18:17.716909 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.716915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.717427 (+ 512us) server_negotiation.cc:407] Connection header received
0812 11:18:17.718055 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.718059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.718115 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.718269 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.719699 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.720390 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.721716 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.721943 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.722046 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.726136 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.726140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.726142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.726173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.727790 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.728463 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.728469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.728471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.728558 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.728916 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.728920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.728921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.729151 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.729266 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.729822 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.729980 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 11:18:17.731470 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.716618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45737 (local address 127.0.0.1:51682)
0812 11:18:17.717182 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.717256 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.717742 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.718328 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.718334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.718991 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.719481 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.719500 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.720551 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.720557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.721569 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.721579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.722156 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.722164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.722334 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.723334 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.723354 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.725937 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.727988 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.728001 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.728017 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.728337 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.728658 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.728662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.728664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.728809 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.729268 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.729279 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.729579 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.729781 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.730203 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":46,"spinlock_wait_cycles":13312,"thread_start_us":225,"threads_started":1}
I20260812 11:18:17.731482 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.715822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.27.109.60:38079)
0812 11:18:17.716468 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.716488 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.716562 (+ 74us) server_negotiation.cc:407] Connection header received
0812 11:18:17.717742 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.717774 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.718073 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.718328 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.719308 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.720613 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.721574 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.722135 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.722336 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.725418 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.725426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.725442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.725472 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.727922 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.728557 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.728564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.728566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.728704 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.729024 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.729028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.729029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.729562 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.729832 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.730106 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.730295 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":414,"thread_start_us":179,"threads_started":1}
I20260812 11:18:17.733234 17228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:17.743707 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.734437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43071 (local address 127.0.0.1:41926)
0812 11:18:17.734580 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.734595 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.734763 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.735150 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.735152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.735325 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.735419 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.735424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.736156 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.736161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.736735 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.736742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.737194 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.737197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.737287 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.737883 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.737898 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.739470 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.741288 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.741292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.741294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.741601 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.742109 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.742114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.742117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.742243 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.743029 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.743033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.743183 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.743354 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.743532 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 11:18:17.743891 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.734081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.27.109.61:43071)
0812 11:18:17.734279 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.734284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.734669 (+ 385us) server_negotiation.cc:407] Connection header received
0812 11:18:17.734925 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.734928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.734973 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.735019 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.735537 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.736046 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.736863 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.737079 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.737188 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.739620 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.739623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.739625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.739648 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.741137 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.741798 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.741805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.741808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.741954 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.742470 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.742476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.742485 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.742808 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.743012 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.743404 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.743586 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:18:17.744993 17228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:18:17.745744 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.734783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.27.109.62:45737)
0812 11:18:17.734967 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.734972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.735136 (+ 164us) server_negotiation.cc:407] Connection header received
0812 11:18:17.735335 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.735338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.735388 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.735477 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.736206 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.736749 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.737693 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.738148 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.738289 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.741283 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.741287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.741290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.741319 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.743411 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.744011 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.744016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.744018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.744145 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.744567 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.744571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.744572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.744911 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.745127 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.745358 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.745518 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:18:17.755723 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.745755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59138)
0812 11:18:17.745952 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.745970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.746054 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.746452 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.746456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.746720 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.746893 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.746905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.748015 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.748017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.748624 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.748636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.749221 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.749225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.749333 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.749903 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.749915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.751782 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.754033 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.754039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.754042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.754244 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.754536 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.754541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.754543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.754623 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.755110 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.755113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.755206 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.755326 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.755511 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 11:18:17.756238 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.746043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.27.109.60:38079)
0812 11:18:17.746170 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.746178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.746195 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:18:17.746236 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.746239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.746286 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.746400 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.747033 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.747900 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.748798 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.749093 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.749187 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.751975 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.751982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.751986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.752026 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.753915 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.754353 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.754359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.754361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.754443 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.754721 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.754724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.754726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.755012 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.755124 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.755374 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.755643 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":44}
I20260812 11:18:17.757745 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.747389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.27.109.61:43071)
0812 11:18:17.747523 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.747526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.747540 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:18:17.747728 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.747730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.747771 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.747843 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.748647 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.749255 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.750120 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.750497 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.750622 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.753715 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.753719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.753721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.753747 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.755383 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.756057 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.756061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.756063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.756170 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.756665 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.756670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.756672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.756952 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.757239 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.757421 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.757581 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:18:17.761493 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: Bootstrap starting.
I20260812 11:18:17.762220 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:18:17.765051 17228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:17.766397 17228 log.cc:824] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:17.767581 16975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } attempt: 3
I20260812 11:18:17.770670 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: No bootstrap required, opened a new log
I20260812 11:18:17.776889 17228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.777323 17228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:17.777395 17228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49afe1c0ad0640828cb18080a8bc3463, State: Initialized, Role: FOLLOWER
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260812 11:18:17.778151 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.764726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.27.109.62:45737)
0812 11:18:17.765672 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.765676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.765690 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:18:17.765742 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.765744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.765789 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.765894 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.767036 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.767887 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.769264 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.769672 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.769747 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.773736 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.773741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.773744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.773786 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.775903 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.776475 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.776479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.776480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.776587 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.776915 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.776921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.776926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.777205 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.777350 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.777802 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.777963 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842}
I20260812 11:18:17.778170 17228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.780136 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.765183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.27.109.61:43071)
0812 11:18:17.765348 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.765353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.765868 (+ 515us) server_negotiation.cc:407] Connection header received
0812 11:18:17.766001 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.766006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.766063 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.766158 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.767215 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.767993 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.769682 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.770023 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.770138 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.772720 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.772725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.772728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.772775 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.775313 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.776049 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.776056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.776059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.776161 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.776553 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.776557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.776559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.776853 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.777024 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.779671 (+ 2647us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.779860 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:18:17.780364 17228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:17.783159 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.784586 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:18:17.785517 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.768119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59160)
0812 11:18:17.768283 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.768322 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.768409 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.769096 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.769100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.769341 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.769434 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.769440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.771031 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.771035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.772015 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.772031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.775066 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.775073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.775217 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.775812 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.775836 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.778257 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.782607 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.782611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.782613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.782963 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.783543 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.783547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.783550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.783703 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.784789 (+ 1086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.784792 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.784937 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.785065 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.785286 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 11:18:17.786855 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.765485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.27.109.60:38079)
0812 11:18:17.765625 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.765630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.770682 (+ 5052us) server_negotiation.cc:407] Connection header received
0812 11:18:17.771013 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.771017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.771072 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.771128 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.772205 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.772881 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.776786 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.777180 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.777321 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.782559 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.782567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.782573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.782603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.784600 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.785209 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.785215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.785219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.785314 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.785672 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.785676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.785678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.785941 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.786063 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.786465 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.786607 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":61}
W20260812 11:18:17.787787 17279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:18:17.787904 17279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:18:17.785845 17107 master_runner.cc:186] Error getting master registration for 127.27.109.62:45737: OK, instance_id { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" instance_seqno: 1786533497112390 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:18:17.790177 17107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:18:17.790732 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.768406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.27.109.60:38079)
0812 11:18:17.768811 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.768820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.768836 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:18:17.768880 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.768883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.768935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.769546 (+ 611us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.769662 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.770704 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.772205 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.773550 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.776601 (+ 3051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.778766 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.778771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.778776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.778806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.781316 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.783128 (+ 1812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.783133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.783136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.783344 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.784093 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.784098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.784100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.784402 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.790341 (+ 5939us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.790436 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.790563 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":312,"thread_start_us":158,"threads_started":1}
I20260812 11:18:17.794278 16975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: Bootstrap starting.
I20260812 11:18:17.797506 16975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:17.798400 16975 log.cc:824] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:17.801613 16975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: No bootstrap required, opened a new log
I20260812 11:18:17.805132 16975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.805399 16975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:17.805487 16975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e859a681ccc4b5a8c023eb0e24c863b, State: Initialized, Role: FOLLOWER
I20260812 11:18:17.806025 16975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.806795 17284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.807106 17284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:18:17.807717 16975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:17.809763 17094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } attempt: 4
W20260812 11:18:17.815918 17301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:18:17.816116 17301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:18:17.813614 16850 master_runner.cc:186] Error getting master registration for 127.27.109.61:43071: OK, instance_id { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" instance_seqno: 1786533497392092 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:18:17.818142 16850 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 11:18:17 dist-test-slave-7z32 krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260812 11:18:17.825058 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.810478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:38079 (local address 127.0.0.1:59172)
0812 11:18:17.811198 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:17.811216 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:17.811351 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:17.811815 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:17.811818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:17.812112 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:17.812248 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.812257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.813468 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.813471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.814371 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:17.814383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.816855 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.816861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:17.817010 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.817759 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:17.817776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:17.819629 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:17.821855 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.821860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.821862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.822101 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.822495 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:17.822499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:17.822501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:17.822604 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:17.823164 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:17.823167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:17.823292 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 11:18:17.824657 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0812 11:18:17.824868 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260812 11:18:17.825672 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:17.810984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.27.109.60:38079)
0812 11:18:17.811136 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:17.811142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:17.811447 (+ 305us) server_negotiation.cc:407] Connection header received
0812 11:18:17.811504 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:17.811508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:17.811564 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:17.811646 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:17.812403 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.813305 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.816258 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:17.816661 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:17.816799 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:17.819841 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:17.819845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:17.819848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:17.819878 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:17.821697 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.822229 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.822250 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.822251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.822390 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:17.822712 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:17.822717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:17.822719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:17.823049 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:17.823175 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:17.825356 (+ 2181us) server_negotiation.cc:299] Negotiation successful
0812 11:18:17.825508 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:18:17.828979 17094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac: Bootstrap starting.
I20260812 11:18:17.831809 17094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:17.832901 17094 log.cc:824] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:17.836537 17094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac: No bootstrap required, opened a new log
I20260812 11:18:17.840202 17094 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.840473 17094 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:17.840536 17094 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0542b67e3134936a2a74bfdb99ecbac, State: Initialized, Role: FOLLOWER
I20260812 11:18:17.841100 17094 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.841696 17307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.841954 17307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:18:17.842674 17094 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: configured and running, proceeding with master startup.
W20260812 11:18:17.850597 17325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:18:17.850819 17325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:18:17.853341 16991 master_runner.cc:263] No leader master found from master e0542b67e3134936a2a74bfdb99ecbac
I20260812 11:18:17.853454 16991 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 11:18:17.857460 16890 tablet.cc:2369] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:18:17.887647 17023 tablet.cc:2369] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:17.894603 17307 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:17.894879 17307 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.896057 17307 leader_election.cc:288] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e859a681ccc4b5a8c023eb0e24c863b (127.27.109.62:45737), 49afe1c0ad0640828cb18080a8bc3463 (127.27.109.60:38079)
I20260812 11:18:17.897037 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0542b67e3134936a2a74bfdb99ecbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49afe1c0ad0640828cb18080a8bc3463" is_pre_election: true
I20260812 11:18:17.897393 17183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0542b67e3134936a2a74bfdb99ecbac in term 0.
I20260812 11:18:17.897853 16926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0542b67e3134936a2a74bfdb99ecbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" is_pre_election: true
I20260812 11:18:17.898161 16926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0542b67e3134936a2a74bfdb99ecbac in term 0.
I20260812 11:18:17.898741 17016 leader_election.cc:302] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49afe1c0ad0640828cb18080a8bc3463, e0542b67e3134936a2a74bfdb99ecbac; no voters:
I20260812 11:18:17.899344 17307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:18:17.899441 17307 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:18:17.899494 17307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:17.900947 17307 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.901340 17307 leader_election.cc:288] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [CANDIDATE]: Term 1 election: Requested vote from peers 1e859a681ccc4b5a8c023eb0e24c863b (127.27.109.62:45737), 49afe1c0ad0640828cb18080a8bc3463 (127.27.109.60:38079)
I20260812 11:18:17.902064 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0542b67e3134936a2a74bfdb99ecbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49afe1c0ad0640828cb18080a8bc3463"
I20260812 11:18:17.902223 17183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:17.902971 16926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0542b67e3134936a2a74bfdb99ecbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e859a681ccc4b5a8c023eb0e24c863b"
I20260812 11:18:17.903128 16926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:17.904619 16926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0542b67e3134936a2a74bfdb99ecbac in term 1.
I20260812 11:18:17.906071 17183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0542b67e3134936a2a74bfdb99ecbac in term 1.
I20260812 11:18:17.906527 17016 leader_election.cc:302] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49afe1c0ad0640828cb18080a8bc3463, e0542b67e3134936a2a74bfdb99ecbac; no voters:
I20260812 11:18:17.906921 17307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:17.907147 17307 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [term 1 LEADER]: Becoming Leader. State: Replica: e0542b67e3134936a2a74bfdb99ecbac, State: Running, Role: LEADER
I20260812 11:18:17.907563 17307 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } }
I20260812 11:18:17.913551 17307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0542b67e3134936a2a74bfdb99ecbac. Latest consensus state: current_term: 1 leader_uuid: "e0542b67e3134936a2a74bfdb99ecbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.913740 17307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:17.914183 17329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:17.915288 17329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:17.925024 17183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [term 1 FOLLOWER]: Refusing update from remote peer e0542b67e3134936a2a74bfdb99ecbac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:17.925858 17327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:17.931026 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0542b67e3134936a2a74bfdb99ecbac. Latest consensus state: current_term: 1 leader_uuid: "e0542b67e3134936a2a74bfdb99ecbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.931209 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:18:17.935204 16926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [term 1 FOLLOWER]: Refusing update from remote peer e0542b67e3134936a2a74bfdb99ecbac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:17.935900 17327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:17.940747 17284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0542b67e3134936a2a74bfdb99ecbac. Latest consensus state: current_term: 1 leader_uuid: "e0542b67e3134936a2a74bfdb99ecbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.940912 17284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [sys.catalog]: This master's current role is: FOLLOWER
W20260812 11:18:17.944988 17141 tablet.cc:2369] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:17.947167 17327 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0542b67e3134936a2a74bfdb99ecbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.947335 17327 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:17.948684 17284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0542b67e3134936a2a74bfdb99ecbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.948835 17284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:18:17.952090 17330 mvcc.cc:205] Tried to move back new op lower bound from 7317641207490506752 to 7317641207444275200. Current Snapshot: MvccSnapshot[applied={T|T < 7317641207490506752}]
I20260812 11:18:17.955590 17329 catalog_manager.cc:1347] Generated new cluster ID: 35b746efe4fe493ea8b302d996616bdc
I20260812 11:18:17.955701 17329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:18:17.967546 17307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0542b67e3134936a2a74bfdb99ecbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.967787 17307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:17.969524 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0542b67e3134936a2a74bfdb99ecbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e859a681ccc4b5a8c023eb0e24c863b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45737 } } peers { permanent_uuid: "e0542b67e3134936a2a74bfdb99ecbac" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43071 } } peers { permanent_uuid: "49afe1c0ad0640828cb18080a8bc3463" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 38079 } } }
I20260812 11:18:17.969686 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:18:17.973963 17333 mvcc.cc:205] Tried to move back new op lower bound from 7317641207490506752 to 7317641207444275200. Current Snapshot: MvccSnapshot[applied={T|T < 7317641207490506752}]
I20260812 11:18:17.974201 17331 mvcc.cc:205] Tried to move back new op lower bound from 7317641207490506752 to 7317641207444275200. Current Snapshot: MvccSnapshot[applied={T|T < 7317641207490506752}]
I20260812 11:18:18.001058 17329 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:18:18.002493 17329 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:18:18.026721 17329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0542b67e3134936a2a74bfdb99ecbac: Generated new TSK 0
I20260812 11:18:18.027830 17329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:18:18.789681 17279 catalog_manager.cc:1259] Loaded cluster ID: 35b746efe4fe493ea8b302d996616bdc
I20260812 11:18:18.789788 17279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: loading cluster ID for follower catalog manager: success
I20260812 11:18:18.791754 17279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: acquiring CA information for follower catalog manager: success
I20260812 11:18:18.793071 17279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49afe1c0ad0640828cb18080a8bc3463: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:18:18.818184 17301 catalog_manager.cc:1259] Loaded cluster ID: 35b746efe4fe493ea8b302d996616bdc
I20260812 11:18:18.818336 17301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: loading cluster ID for follower catalog manager: success
I20260812 11:18:18.821002 17301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: acquiring CA information for follower catalog manager: success
I20260812 11:18:18.823112 17301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e859a681ccc4b5a8c023eb0e24c863b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:18:19.107504 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 16850
I20260812 11:18:19.147238 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 16991
I20260812 11:18:19.183284 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 17107
2026-08-12T11:18:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3935 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:18:19 dist-test-slave-7z32 krb5kdc[17354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:18:19 dist-test-slave-7z32 krb5kdc[17354](info): set up 2 sockets
Aug 12 11:18:19 dist-test-slave-7z32 krb5kdc[17354](info): commencing operation
krb5kdc: starting...
W20260812 11:18:20.566617 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.339s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:18:20 dist-test-slave-7z32 krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:18:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:18:20.751466 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39025
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:20.917055 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:20.917654 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:20.917757 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:20.925246 17376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:20.925460 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:20.925505 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:20.925540 17376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:20.925581 17376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:20.935154 17376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:20.937152 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:20.939107 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:20.952452 17395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:20.952471 17386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:20.952443 17389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:20.953073 17376 server_base.cc:1034] running on GCE node
I20260812 11:18:20.953879 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:20.955482 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:20.956892 17376 hybrid_clock.cc:648] HybridClock initialized: now 1786533500956840 us; error 77 us; skew 500 ppm
Aug 12 11:18:20 dist-test-slave-7z32 krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533500, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:20.978332 17376 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:20.979878 17376 webserver.cc:460] Webserver started at http://127.27.109.62:39805/ using document root <none> and password file <none>
I20260812 11:18:20.980971 17376 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:20.981114 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:20.981616 17376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:20.984875 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "efc71da4e8e04069a1bd44300d2a3a0f"
format_stamp: "Formatted at 2026-08-12 11:18:20 on dist-test-slave-7z32"
server_key: "a705543ddc2db10b3c6dc6f2816ceb7b"
server_key_iv: "2771169e93abf8af197e2f2e1b6d89ea"
server_key_version: "encryptionkey@0"
I20260812 11:18:20.985685 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "efc71da4e8e04069a1bd44300d2a3a0f"
format_stamp: "Formatted at 2026-08-12 11:18:20 on dist-test-slave-7z32"
server_key: "a705543ddc2db10b3c6dc6f2816ceb7b"
server_key_iv: "2771169e93abf8af197e2f2e1b6d89ea"
server_key_version: "encryptionkey@0"
I20260812 11:18:20.991494 17376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 11:18:20.995959 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:20.998492 17376 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:18:20.998708 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "efc71da4e8e04069a1bd44300d2a3a0f"
format_stamp: "Formatted at 2026-08-12 11:18:20 on dist-test-slave-7z32"
server_key: "a705543ddc2db10b3c6dc6f2816ceb7b"
server_key_iv: "2771169e93abf8af197e2f2e1b6d89ea"
server_key_version: "encryptionkey@0"
I20260812 11:18:20.999001 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:21.036999 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:21.042120 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:21.042622 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:21.054677 17376 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39025
I20260812 11:18:21.054944 17488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39025 every 8 connection(s)
I20260812 11:18:21.056027 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:21.060343 17489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:21.065330 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 17376
I20260812 11:18:21.065517 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:18:21.065881 28084 external_mini_cluster.cc:1428] Setting key 8d2f7e17f6079b211647ecd8ab46c151
I20260812 11:18:21.069332 17489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f: Bootstrap starting.
I20260812 11:18:21.072993 17489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:21.074280 17489 log.cc:824] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:21.078073 17489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f: No bootstrap required, opened a new log
Aug 12 11:18:21 dist-test-slave-7z32 krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:21.084225 17489 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39025 } }
I20260812 11:18:21.084828 17489 raft_consensus.cc:374] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:21.084929 17489 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc71da4e8e04069a1bd44300d2a3a0f, State: Initialized, Role: FOLLOWER
I20260812 11:18:21.085647 17489 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39025 } }
I20260812 11:18:21.085840 17489 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:18:21.085964 17489 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:18:21.086100 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:21.067928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.27.109.62:39025)
0812 11:18:21.068491 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:21.068511 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:21.068559 (+ 48us) server_negotiation.cc:407] Connection header received
0812 11:18:21.069554 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:21.069590 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:21.070034 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:21.070415 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:21.071705 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:21.072939 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:21.074443 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:21.075235 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:21.075444 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:21.079284 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:21.079294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:21.079325 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:21.079386 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:21.081635 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:21.082419 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:21.082425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:21.082432 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:21.082587 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:21.083056 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:21.083060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:21.083061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:21.083620 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:21.083817 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:21.084369 (+ 552us) server_negotiation.cc:299] Negotiation successful
0812 11:18:21.084813 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":189,"threads_started":1}
I20260812 11:18:21.086135 17489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:21.087993 17489 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39025 } }
I20260812 11:18:21.088526 17489 leader_election.cc:302] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc71da4e8e04069a1bd44300d2a3a0f; no voters:
I20260812 11:18:21.089084 17489 leader_election.cc:288] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:18:21.089180 17496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:21.089491 17496 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [term 1 LEADER]: Becoming Leader. State: Replica: efc71da4e8e04069a1bd44300d2a3a0f, State: Running, Role: LEADER
I20260812 11:18:21.090054 17496 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39025 } }
I20260812 11:18:21.090255 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:18:21.091029 17489 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:21.092533 17500 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc71da4e8e04069a1bd44300d2a3a0f. Latest consensus state: current_term: 1 leader_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39025 } } }
I20260812 11:18:21.092736 17500 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:21.093170 17498 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc71da4e8e04069a1bd44300d2a3a0f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39025 } } }
I20260812 11:18:21.093271 17512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:21.093386 17498 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:21.097519 17512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:21.104453 17512 catalog_manager.cc:1347] Generated new cluster ID: 7f622bd8b59c49a5abdc757026083ef4
I20260812 11:18:21.104578 17512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:18:21.117377 17512 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:18:21.118736 17512 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:18:21.126585 17512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efc71da4e8e04069a1bd44300d2a3a0f: Generated new TSK 0
I20260812 11:18:21.127774 17512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:18:21.150993 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 17376
2026-08-12T11:18:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1953 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:18:21 dist-test-slave-7z32 krb5kdc[17536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:18:21 dist-test-slave-7z32 krb5kdc[17536](info): set up 2 sockets
Aug 12 11:18:21 dist-test-slave-7z32 krb5kdc[17536](info): commencing operation
krb5kdc: starting...
W20260812 11:18:22.599013 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.418s user 0.002s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:18:22 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:18:22.772809 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46227
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:22.903504 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:22.903846 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:22.903896 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:22.908692 17556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:22.908816 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:22.908862 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:22.908936 17556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:22.908977 17556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:22.914942 17556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:22.916273 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:22.917490 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:22.928367 17569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:22.928368 17567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:22.928463 17556 server_base.cc:1034] running on GCE node
W20260812 11:18:22.928390 17571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:22.929559 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:22.931643 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:22.932895 17556 hybrid_clock.cc:648] HybridClock initialized: now 1786533502932854 us; error 71 us; skew 500 ppm
Aug 12 11:18:22 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533502, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:22.937569 17556 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:22.939194 17556 webserver.cc:460] Webserver started at http://127.27.109.62:45709/ using document root <none> and password file <none>
I20260812 11:18:22.940004 17556 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:22.940101 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:22.940346 17556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:22.943547 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "32cf5ccd919648799de1c6dafc954386"
format_stamp: "Formatted at 2026-08-12 11:18:22 on dist-test-slave-7z32"
server_key: "aabc6bf5ac61e04a2b22a33557de4a24"
server_key_iv: "b87cc90ceccb195e2bf7d1aa5b28998b"
server_key_version: "encryptionkey@0"
I20260812 11:18:22.944310 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "32cf5ccd919648799de1c6dafc954386"
format_stamp: "Formatted at 2026-08-12 11:18:22 on dist-test-slave-7z32"
server_key: "aabc6bf5ac61e04a2b22a33557de4a24"
server_key_iv: "b87cc90ceccb195e2bf7d1aa5b28998b"
server_key_version: "encryptionkey@0"
I20260812 11:18:22.949954 17556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:18:22.953811 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:22.956020 17556 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 11:18:22.956281 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "32cf5ccd919648799de1c6dafc954386"
format_stamp: "Formatted at 2026-08-12 11:18:22 on dist-test-slave-7z32"
server_key: "aabc6bf5ac61e04a2b22a33557de4a24"
server_key_iv: "b87cc90ceccb195e2bf7d1aa5b28998b"
server_key_version: "encryptionkey@0"
I20260812 11:18:22.956485 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:22.971375 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:22.976539 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:22.976876 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:22.988129 17556 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:46227
I20260812 11:18:22.988497 17674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:46227 every 8 connection(s)
I20260812 11:18:22.989665 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:22.993461 17675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:22.994369 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 17556
I20260812 11:18:22.994513 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:18:22.994988 28084 external_mini_cluster.cc:1428] Setting key 809641df864bca600108891f7df4600e
I20260812 11:18:23.001806 17675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Bootstrap starting.
I20260812 11:18:23.005075 17675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:23.006057 17675 log.cc:824] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:18:23 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:23.009553 17675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: No bootstrap required, opened a new log
I20260812 11:18:23.015625 17675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:23.016039 17675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:23.016084 17675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cf5ccd919648799de1c6dafc954386, State: Initialized, Role: FOLLOWER
I20260812 11:18:23.016654 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:22.996999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.27.109.62:46227)
0812 11:18:22.997700 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:22.997725 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:22.997782 (+ 57us) server_negotiation.cc:407] Connection header received
0812 11:18:22.998770 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:22.998797 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:22.999446 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:22.999933 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:23.001049 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.002420 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.003667 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.004359 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.004600 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:23.008863 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:23.008873 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:23.008898 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:23.008950 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:23.011584 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:23.012612 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:23.012621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:23.012629 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:23.012860 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:23.013263 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:23.013265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:23.013266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:23.013948 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:23.014184 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:23.014618 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 11:18:23.015242 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":204,"threads_started":1}
I20260812 11:18:23.016737 17675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:23.016955 17675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:18:23.017071 17675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:18:23.017256 17675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:23.018859 17675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:23.019332 17675 leader_election.cc:302] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32cf5ccd919648799de1c6dafc954386; no voters:
I20260812 11:18:23.019827 17675 leader_election.cc:288] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:18:23.020119 17683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:23.020547 17683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 LEADER]: Becoming Leader. State: Replica: 32cf5ccd919648799de1c6dafc954386, State: Running, Role: LEADER
I20260812 11:18:23.021054 17683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:23.021520 17675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:23.023648 17685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32cf5ccd919648799de1c6dafc954386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } } }
I20260812 11:18:23.023630 17686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32cf5ccd919648799de1c6dafc954386. Latest consensus state: current_term: 1 leader_uuid: "32cf5ccd919648799de1c6dafc954386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } } }
I20260812 11:18:23.023852 17686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:23.023852 17685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: This master's current role is: LEADER
W20260812 11:18:23.029232 17707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:18:23.029421 17707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:18:23.029527 17703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:23.030627 17703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:23.038777 17703 catalog_manager.cc:1347] Generated new cluster ID: cfe457661ebe4a4aa2388bd1663406e7
I20260812 11:18:23.038952 17703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:18:23.073765 17703 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:18:23.075311 17703 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:18:23.082239 17703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Generated new TSK 0
I20260812 11:18:23.083737 17703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:18:23.139926 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46227
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:23.293735 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:23.294123 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:23.294178 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:23.300175 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:23.300454 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:23.300571 17719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:18:23.307485 17719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:23.309075 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:23.310456 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:23.324641 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:23.324640 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:23.325038 17729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:23.325327 17719 server_base.cc:1034] running on GCE node
I20260812 11:18:23.325906 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:23.326777 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:23.328073 17719 hybrid_clock.cc:648] HybridClock initialized: now 1786533503328024 us; error 66 us; skew 500 ppm
Aug 12 11:18:23 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533503, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:23.332248 17719 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:18:23.333639 17719 webserver.cc:460] Webserver started at http://127.27.109.1:33511/ using document root <none> and password file <none>
I20260812 11:18:23.334579 17719 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:23.334692 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:23.335075 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:23.338039 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "1421b1e4e4c54c3e81ff323d791e0555"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "06d3c7fe16e10389c3ac438cb59191c8"
server_key_iv: "ab7700a66849c721871e9b192b1b9240"
server_key_version: "encryptionkey@0"
I20260812 11:18:23.338871 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "1421b1e4e4c54c3e81ff323d791e0555"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "06d3c7fe16e10389c3ac438cb59191c8"
server_key_iv: "ab7700a66849c721871e9b192b1b9240"
server_key_version: "encryptionkey@0"
I20260812 11:18:23.344645 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:18:23.348778 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:23.351030 17719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:18:23.351297 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "1421b1e4e4c54c3e81ff323d791e0555"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "06d3c7fe16e10389c3ac438cb59191c8"
server_key_iv: "ab7700a66849c721871e9b192b1b9240"
server_key_version: "encryptionkey@0"
I20260812 11:18:23.351529 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:23.415236 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:23.420720 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:23.421056 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:23.422026 17719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:23.423547 17719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:23.423619 17719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:23.423671 17719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:23.423687 17719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:23.437902 17719 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34219
I20260812 11:18:23.438032 17947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34219 every 8 connection(s)
I20260812 11:18:23.439553 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:18:23.443823 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 17719
I20260812 11:18:23.444024 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:18:23.444486 28084 external_mini_cluster.cc:1428] Setting key 2cf9edd43ccb29a3e98669a69fbbbbe2
Aug 12 11:18:23 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533503, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:18:23.460256 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:23.442370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:55819 (local address 127.27.109.62:46227)
0812 11:18:23.442699 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:23.442704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:23.444195 (+ 1491us) server_negotiation.cc:407] Connection header received
0812 11:18:23.446089 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:23.446093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:23.446159 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:23.446296 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:23.448040 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.449017 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.450617 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.451107 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.451269 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:23.454926 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:23.454931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:23.454933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:23.454972 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:23.457019 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:23.458058 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:23.458063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:23.458065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:23.458163 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:23.458611 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:23.458615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:23.458617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:23.458955 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:23.459103 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:23.459850 (+ 747us) server_negotiation.cc:299] Negotiation successful
0812 11:18:23.460046 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260812 11:18:23.461586 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:23.443039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:55819)
0812 11:18:23.443816 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:23.443918 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:23.445769 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:23.446628 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:23.446636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:23.447242 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:23.447826 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:23.447843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.449224 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.449227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:23.450437 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:23.450449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.451332 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.451336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:23.451521 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:23.452249 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:23.452274 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:23.454590 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:23.457233 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:23.457243 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:23.457266 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:23.457912 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:23.458297 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:23.458301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:23.458303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:23.458494 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:23.459098 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:23.459109 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:23.459512 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 11:18:23.459795 (+ 283us) client_negotiation.cc:240] Negotiation successful
0812 11:18:23.460310 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":211,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:18:23.463593 17948 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46227
I20260812 11:18:23.464020 17948 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:23.464722 17948 heartbeater.cc:511] Master 127.27.109.62:46227 requested a full tablet report, sending...
I20260812 11:18:23.466792 17608 ts_manager.cc:194] Registered new tserver with Master: 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219)
I20260812 11:18:23.468633 17608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:55819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:18:23.487883 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46227
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:23.632220 17959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:23.632555 17959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:23.632602 17959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:23.636943 17959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:23.637039 17959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:23.637104 17959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:18:23.643487 17959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:23.644820 17959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:23.646171 17959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:23.658775 17969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:23.658928 17967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:23.658785 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:23.659227 17959 server_base.cc:1034] running on GCE node
I20260812 11:18:23.659934 17959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:23.660851 17959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:23.662217 17959 hybrid_clock.cc:648] HybridClock initialized: now 1786533503662190 us; error 61 us; skew 500 ppm
Aug 12 11:18:23 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533503, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:23.666124 17959 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:18:23.667573 17959 webserver.cc:460] Webserver started at http://127.27.109.2:43435/ using document root <none> and password file <none>
I20260812 11:18:23.668324 17959 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:23.668444 17959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:23.668730 17959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:23.671788 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "4ec79a765d4d4a16b3ec7be780fec4e5"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "dac6c04f14125a00a44fd6b957a35d16"
server_key_iv: "0ce1951231a210ada0213e6faedd4660"
server_key_version: "encryptionkey@0"
I20260812 11:18:23.672757 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "4ec79a765d4d4a16b3ec7be780fec4e5"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "dac6c04f14125a00a44fd6b957a35d16"
server_key_iv: "0ce1951231a210ada0213e6faedd4660"
server_key_version: "encryptionkey@0"
I20260812 11:18:23.679164 17959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:18:23.683350 17981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:23.685631 17959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:18:23.685894 17959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "4ec79a765d4d4a16b3ec7be780fec4e5"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "dac6c04f14125a00a44fd6b957a35d16"
server_key_iv: "0ce1951231a210ada0213e6faedd4660"
server_key_version: "encryptionkey@0"
I20260812 11:18:23.686090 17959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:23.734107 17959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:23.737923 17959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:23.738124 17959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:23.738759 17959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:23.739857 17959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:23.739943 17959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:23.740017 17959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:23.740065 17959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:23.751565 17959 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:42839
I20260812 11:18:23.751658 18179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:42839 every 8 connection(s)
I20260812 11:18:23.752712 17959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:18:23.761577 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 17959
I20260812 11:18:23.761821 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:18:23.762215 28084 external_mini_cluster.cc:1428] Setting key f0ecea653e38702a8e65fc937d89773c
Aug 12 11:18:23 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533503, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:23.772043 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:23.755589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:53699 (local address 127.27.109.62:46227)
0812 11:18:23.755810 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:23.755816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:23.756783 (+ 967us) server_negotiation.cc:407] Connection header received
0812 11:18:23.758342 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:23.758347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:23.758430 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:23.758632 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:23.760257 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.761256 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.763365 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.763635 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.763756 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:23.767723 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:23.767728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:23.767731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:23.767764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:23.769446 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:23.770116 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:23.770124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:23.770125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:23.770206 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:23.770613 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:23.770616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:23.770620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:23.770947 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:23.771075 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:23.771679 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 11:18:23.771837 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 11:18:23.772971 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:23.755957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:53699)
0812 11:18:23.756554 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:23.756614 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:23.758051 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:23.758902 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:23.758911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:23.759420 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:23.760017 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:23.760034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.761486 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.761489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:23.762925 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:23.762939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:23.763897 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:23.763902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:23.764193 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:23.765047 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:23.765068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:23.767435 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:23.769631 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:23.769640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:23.769664 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:23.769988 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:23.770317 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:23.770321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:23.770323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:23.770496 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:23.771075 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:23.771083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:23.771400 (+ 317us) client_negotiation.cc:769] Sending connection context
0812 11:18:23.771617 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 11:18:23.772009 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":182,"threads_started":1}
I20260812 11:18:23.774603 18181 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46227
I20260812 11:18:23.775136 18181 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:23.776136 18181 heartbeater.cc:511] Master 127.27.109.62:46227 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:18:23.777941 17608 ts_manager.cc:194] Registered new tserver with Master: 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839)
I20260812 11:18:23.778800 17608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:53699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:18:23.808183 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46227
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:23.969827 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:23.970291 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:23.970348 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:23.975479 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:23.975628 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:23.975705 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:18:23.983124 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:23.984885 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:23.986917 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:24.003437 18201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:24.003620 18198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:24.003408 18197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:24.003990 18187 server_base.cc:1034] running on GCE node
I20260812 11:18:24.004714 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:24.005733 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:24.007061 18187 hybrid_clock.cc:648] HybridClock initialized: now 1786533504006986 us; error 97 us; skew 500 ppm
Aug 12 11:18:24 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:24.011372 18187 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:18:24.012959 18187 webserver.cc:460] Webserver started at http://127.27.109.3:43673/ using document root <none> and password file <none>
I20260812 11:18:24.013796 18187 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:24.013918 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:24.014240 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:24.017254 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "e5131b92bf6545079aff20123940b83d"
format_stamp: "Formatted at 2026-08-12 11:18:24 on dist-test-slave-7z32"
server_key: "a0a0b951d55e6fe433de73ee748f5781"
server_key_iv: "f3ed40abbcfb0375882a3709947ae503"
server_key_version: "encryptionkey@0"
I20260812 11:18:24.018150 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "e5131b92bf6545079aff20123940b83d"
format_stamp: "Formatted at 2026-08-12 11:18:24 on dist-test-slave-7z32"
server_key: "a0a0b951d55e6fe433de73ee748f5781"
server_key_iv: "f3ed40abbcfb0375882a3709947ae503"
server_key_version: "encryptionkey@0"
I20260812 11:18:24.024745 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 11:18:24.029204 18212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:24.031656 18187 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:18:24.031885 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "e5131b92bf6545079aff20123940b83d"
format_stamp: "Formatted at 2026-08-12 11:18:24 on dist-test-slave-7z32"
server_key: "a0a0b951d55e6fe433de73ee748f5781"
server_key_iv: "f3ed40abbcfb0375882a3709947ae503"
server_key_version: "encryptionkey@0"
I20260812 11:18:24.032076 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:24.056654 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:24.060879 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:24.061158 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:24.061877 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:24.062999 18187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:24.063079 18187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:24.063153 18187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:24.063210 18187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:24.074977 18187 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:44629
I20260812 11:18:24.075001 18409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:44629 every 8 connection(s)
I20260812 11:18:24.076040 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:18:24.080926 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 18187
I20260812 11:18:24.081051 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:18:24.081388 28084 external_mini_cluster.cc:1428] Setting key 8a8a937bff7445ce19f459c45ea57dab
Aug 12 11:18:24 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:24.096711 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.078761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38103 (local address 127.27.109.62:46227)
0812 11:18:24.079021 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.079027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.080105 (+ 1078us) server_negotiation.cc:407] Connection header received
0812 11:18:24.082316 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.082320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.082387 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.082511 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:24.084833 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.085491 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.087081 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.087351 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.087511 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.091292 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:24.091297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:24.091299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:24.091335 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:24.093236 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:24.094035 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:24.094044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:24.094046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:24.094146 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:24.094652 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:24.094656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:24.094657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:24.095193 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:24.095352 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.096233 (+ 881us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.096486 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 11:18:24.098105 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.079172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:38103)
0812 11:18:24.079844 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.079920 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:24.081975 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:24.082884 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:24.082897 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:24.083649 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:24.084626 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.084670 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.085676 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.085680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.086904 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.086923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.087575 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.087580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.087844 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.088737 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:24.088759 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:24.091004 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:24.093443 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:24.093454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:24.093486 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:24.093867 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:24.094279 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:24.094283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:24.094286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:24.094502 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:24.095349 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:24.095360 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:24.095776 (+ 416us) client_negotiation.cc:769] Sending connection context
0812 11:18:24.096072 (+ 296us) client_negotiation.cc:240] Negotiation successful
0812 11:18:24.096624 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":127,"threads_started":1}
I20260812 11:18:24.100032 18412 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46227
I20260812 11:18:24.100448 18412 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:24.101305 18412 heartbeater.cc:511] Master 127.27.109.62:46227 requested a full tablet report, sending...
I20260812 11:18:24.102990 17608 ts_manager.cc:194] Registered new tserver with Master: e5131b92bf6545079aff20123940b83d (127.27.109.3:44629)
I20260812 11:18:24.103734 17608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:38103
I20260812 11:18:24.110302 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:18:24.123476 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.112962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.27.109.62:46227)
0812 11:18:24.113201 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.113207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.113491 (+ 284us) server_negotiation.cc:407] Connection header received
0812 11:18:24.113681 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.113689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.113736 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.113830 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:24.114686 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.115333 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.116224 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.116457 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.116573 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.118476 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:24.118486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:24.118488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:24.118519 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:24.120580 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:24.121286 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:24.121291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:24.121292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:24.121446 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:24.122025 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:24.122028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:24.122030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:24.122385 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:24.122535 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.123034 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.123231 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 11:18:24.127616 17608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:18:24.131044 17608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:18:24.152498 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.144441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:44629 (local address 127.0.0.1:58052)
0812 11:18:24.145185 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.145248 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:24.145428 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:24.146484 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:24.146490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:24.146520 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:24.146685 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.146694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.148749 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.148753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.149952 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.149967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.151464 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.151469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.152032 (+ 563us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.152056 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:18:24.152142 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 11:18:24.152283 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":433,"thread_start_us":159,"threads_started":1}
I20260812 11:18:24.152570 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.144959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.27.109.2:42839)
0812 11:18:24.145583 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.145614 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.145650 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:18:24.145778 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.145786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.146073 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.146278 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:24.146742 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.148152 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.150384 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.151276 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.152026 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.152207 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.152274 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.152402 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":207,"threads_started":1}
I20260812 11:18:24.152824 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.144626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.27.109.3:44629)
0812 11:18:24.145989 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.146002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.146027 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:18:24.146110 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.146115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.146300 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.146880 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:24.147304 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.148581 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.150170 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.151276 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.151659 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.151888 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.152402 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.152629 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":111,"threads_started":1}
I20260812 11:18:24.153321 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.145098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:42839 (local address 127.0.0.1:55190)
0812 11:18:24.145400 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.145415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:24.145491 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:24.146297 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:24.146372 (+ 75us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:24.146399 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:24.146563 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.146572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.148388 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.148393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.150193 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.150209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.151464 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.151469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.151675 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.151738 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 11:18:24.151879 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 11:18:24.153171 (+ 1292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":1289,"thread_start_us":139,"threads_started":1}
I20260812 11:18:24.153860 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.145751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.27.109.1:34219)
0812 11:18:24.146229 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.146237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.146262 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:18:24.146341 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.146346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.146570 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.146725 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:24.147033 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.148127 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.151203 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.152459 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.152609 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.152795 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.153545 (+ 750us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.153693 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":138,"threads_started":1}
I20260812 11:18:24.155113 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.145545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60800)
0812 11:18:24.145890 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.145907 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:24.145998 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:24.146734 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:24.146737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:24.146755 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:24.146898 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.146904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.148263 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.148269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.150193 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.150209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.152893 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.152899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.153206 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.153243 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:18:24.153347 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 11:18:24.154967 (+ 1620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":291,"thread_start_us":104,"threads_started":1}
I20260812 11:18:24.157639 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 803c8d29e95740b9b9930aa703225909 (DEFAULT_TABLE table=test-table [id=f9f5273de861492894b42230e7257574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:24.157785 18068 tablet_service.cc:1467] Processing CreateTablet for tablet 803c8d29e95740b9b9930aa703225909 (DEFAULT_TABLE table=test-table [id=f9f5273de861492894b42230e7257574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:24.159075 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803c8d29e95740b9b9930aa703225909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:24.158325 17836 tablet_service.cc:1467] Processing CreateTablet for tablet 803c8d29e95740b9b9930aa703225909 (DEFAULT_TABLE table=test-table [id=f9f5273de861492894b42230e7257574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:24.160142 18068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803c8d29e95740b9b9930aa703225909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:24.160175 17836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803c8d29e95740b9b9930aa703225909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:24.168282 18445 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Bootstrap starting.
I20260812 11:18:24.170569 18445 tablet_bootstrap.cc:654] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:24.171433 18447 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap starting.
I20260812 11:18:24.172201 18445 log.cc:824] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:24.173233 18448 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Bootstrap starting.
I20260812 11:18:24.174523 18447 tablet_bootstrap.cc:654] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:24.175523 18447 log.cc:824] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:24.177413 18448 tablet_bootstrap.cc:654] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:24.177917 18445 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: No bootstrap required, opened a new log
I20260812 11:18:24.178201 18445 ts_tablet_manager.cc:1397] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260812 11:18:24.178757 18447 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: No bootstrap required, opened a new log
I20260812 11:18:24.178879 18448 log.cc:824] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:24.179029 18447 ts_tablet_manager.cc:1397] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260812 11:18:24.181901 18448 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: No bootstrap required, opened a new log
I20260812 11:18:24.182458 18448 ts_tablet_manager.cc:1397] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260812 11:18:24.183540 18445 raft_consensus.cc:348] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.183971 18445 raft_consensus.cc:374] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:24.184051 18445 raft_consensus.cc:729] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5131b92bf6545079aff20123940b83d, State: Initialized, Role: FOLLOWER
I20260812 11:18:24.184481 18447 raft_consensus.cc:348] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.184772 18447 raft_consensus.cc:374] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:24.184844 18447 raft_consensus.cc:729] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1421b1e4e4c54c3e81ff323d791e0555, State: Initialized, Role: FOLLOWER
I20260812 11:18:24.185186 18445 consensus_queue.cc:260] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.185243 18447 consensus_queue.cc:260] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.186707 18447 ts_tablet_manager.cc:1428] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260812 11:18:24.186707 18445 ts_tablet_manager.cc:1428] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:18:24.186764 17948 heartbeater.cc:503] Master 127.27.109.62:46227 was elected leader, sending a full tablet report...
I20260812 11:18:24.186806 18412 heartbeater.cc:503] Master 127.27.109.62:46227 was elected leader, sending a full tablet report...
I20260812 11:18:24.190146 18448 raft_consensus.cc:348] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.190526 18448 raft_consensus.cc:374] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:24.190606 18448 raft_consensus.cc:729] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec79a765d4d4a16b3ec7be780fec4e5, State: Initialized, Role: FOLLOWER
I20260812 11:18:24.191366 18448 consensus_queue.cc:260] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
W20260812 11:18:24.191615 17949 tablet.cc:2369] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:24.192663 18181 heartbeater.cc:503] Master 127.27.109.62:46227 was elected leader, sending a full tablet report...
I20260812 11:18:24.192883 18448 ts_tablet_manager.cc:1428] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260812 11:18:24.217096 18458 raft_consensus.cc:482] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:24.217448 18458 raft_consensus.cc:504] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.218961 18458 leader_election.cc:288] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5131b92bf6545079aff20123940b83d (127.27.109.3:44629), 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219)
I20260812 11:18:24.225267 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.219360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.27.109.2:54555)
0812 11:18:24.219655 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.219673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:24.219813 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:24.220229 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:24.220233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:24.220314 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:24.220562 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.220575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.221685 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.221690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.223199 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.223213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.224340 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.224344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.224488 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.224526 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 11:18:24.224916 (+ 390us) client_negotiation.cc:240] Negotiation successful
0812 11:18:24.225062 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":38,"spinlock_wait_cycles":2816}
I20260812 11:18:24.225348 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.219631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:54555 (local address 127.27.109.1:34219)
0812 11:18:24.219912 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.219917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.219936 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:18:24.220007 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.220010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.220065 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.220160 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:24.220721 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.221465 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.223407 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.224119 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.224243 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.224294 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.225080 (+ 786us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.225210 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260812 11:18:24.225618 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.219690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:56147 (local address 127.27.109.3:44629)
0812 11:18:24.219923 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.219930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.220307 (+ 377us) server_negotiation.cc:407] Connection header received
0812 11:18:24.220475 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.220479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.220546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.220679 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:24.221138 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.222022 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.223820 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.224580 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.224899 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.224967 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.225322 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.225465 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:18:24.226147 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.219554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:44629 (local address 127.27.109.2:56147)
0812 11:18:24.220178 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.220195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:24.220335 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:24.220721 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:24.220724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:24.220750 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:24.220901 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.220966 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.222202 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.222205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.223602 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:24.223617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.224964 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.224971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:24.225107 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.225143 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:18:24.225884 (+ 741us) client_negotiation.cc:240] Negotiation successful
0812 11:18:24.225987 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":5,"spinlock_wait_cycles":2944,"thread_start_us":242,"threads_started":1}
I20260812 11:18:24.226743 17877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c8d29e95740b9b9930aa703225909" candidate_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1421b1e4e4c54c3e81ff323d791e0555" is_pre_election: true
I20260812 11:18:24.227156 17877 raft_consensus.cc:2454] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec79a765d4d4a16b3ec7be780fec4e5 in term 0.
I20260812 11:18:24.227123 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c8d29e95740b9b9930aa703225909" candidate_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5131b92bf6545079aff20123940b83d" is_pre_election: true
I20260812 11:18:24.227632 18330 raft_consensus.cc:2454] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec79a765d4d4a16b3ec7be780fec4e5 in term 0.
I20260812 11:18:24.227914 17985 leader_election.cc:302] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1421b1e4e4c54c3e81ff323d791e0555, 4ec79a765d4d4a16b3ec7be780fec4e5; no voters:
I20260812 11:18:24.228379 18458 raft_consensus.cc:2781] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:18:24.228497 18458 raft_consensus.cc:482] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:18:24.228550 18458 raft_consensus.cc:3037] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:24.229916 18458 raft_consensus.cc:504] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.230376 18458 leader_election.cc:288] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [CANDIDATE]: Term 1 election: Requested vote from peers e5131b92bf6545079aff20123940b83d (127.27.109.3:44629), 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219)
I20260812 11:18:24.231078 17877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c8d29e95740b9b9930aa703225909" candidate_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1421b1e4e4c54c3e81ff323d791e0555"
I20260812 11:18:24.231046 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c8d29e95740b9b9930aa703225909" candidate_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5131b92bf6545079aff20123940b83d"
I20260812 11:18:24.231227 17877 raft_consensus.cc:3037] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:24.231318 18330 raft_consensus.cc:3037] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:24.232666 17877 raft_consensus.cc:2454] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec79a765d4d4a16b3ec7be780fec4e5 in term 1.
I20260812 11:18:24.233163 18330 raft_consensus.cc:2454] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec79a765d4d4a16b3ec7be780fec4e5 in term 1.
I20260812 11:18:24.233196 17985 leader_election.cc:302] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1421b1e4e4c54c3e81ff323d791e0555, 4ec79a765d4d4a16b3ec7be780fec4e5; no voters:
I20260812 11:18:24.233496 18458 raft_consensus.cc:2781] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:24.233886 18458 raft_consensus.cc:686] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 LEADER]: Becoming Leader. State: Replica: 4ec79a765d4d4a16b3ec7be780fec4e5, State: Running, Role: LEADER
I20260812 11:18:24.234290 18458 consensus_queue.cc:237] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:24.239104 17605 catalog_manager.cc:5495] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } health_report { overall_health: UNKNOWN } } }
W20260812 11:18:24.254604 18182 tablet.cc:2369] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:24.259743 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.254230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.27.109.2:42839)
0812 11:18:24.254461 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:24.254466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:24.254490 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:18:24.254630 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:24.254633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:24.254759 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:24.255015 (+ 256us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:18:24.255361 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.255936 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.257066 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:24.257308 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:24.257470 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:24.258180 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:18:24.259028 (+ 848us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:18:24.259168 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:24.259398 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 11:18:24.259491 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 11:18:24.270547 17877 raft_consensus.cc:1261] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Refusing update from remote peer 4ec79a765d4d4a16b3ec7be780fec4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:24.270849 18330 raft_consensus.cc:1261] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 1 FOLLOWER]: Refusing update from remote peer 4ec79a765d4d4a16b3ec7be780fec4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:24.272245 18458 consensus_queue.cc:1035] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:24.272646 18460 consensus_queue.cc:1035] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:24.282097 18466 mvcc.cc:205] Tried to move back new op lower bound from 7317641233480101888 to 7317641233347592192. Current Snapshot: MvccSnapshot[applied={T|T < 7317641233480101888}]
I20260812 11:18:24.285672 18469 mvcc.cc:205] Tried to move back new op lower bound from 7317641233480101888 to 7317641233347592192. Current Snapshot: MvccSnapshot[applied={T|T < 7317641233480101888}]
I20260812 11:18:24.295786 17608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:24.296094 17608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:24.299664 17608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Sending DeleteTablet for 3 replicas of tablet 803c8d29e95740b9b9930aa703225909
I20260812 11:18:24.301008 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 17719
I20260812 11:18:24.301064 18068 tablet_service.cc:1514] Processing DeleteTablet for tablet 803c8d29e95740b9b9930aa703225909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:24 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:55190
I20260812 11:18:24.301229 18299 tablet_service.cc:1514] Processing DeleteTablet for tablet 803c8d29e95740b9b9930aa703225909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:24 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:58052
I20260812 11:18:24.314324 18489 tablet_replica.cc:330] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: stopping tablet replica
I20260812 11:18:24.315177 18489 raft_consensus.cc:2229] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:18:24.316072 18489 raft_consensus.cc:2258] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:24.325516 18489 ts_tablet_manager.cc:1905] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:24.332209 18489 ts_tablet_manager.cc:1918] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:24.332429 18489 log.cc:1196] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/803c8d29e95740b9b9930aa703225909
I20260812 11:18:24.332906 18489 ts_tablet_manager.cc:1939] T 803c8d29e95740b9b9930aa703225909 P 4ec79a765d4d4a16b3ec7be780fec4e5: Deleting consensus metadata
I20260812 11:18:24.335419 17590 catalog_manager.cc:4841] TS 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839): tablet 803c8d29e95740b9b9930aa703225909 (table test-table [id=f9f5273de861492894b42230e7257574]) successfully deleted
I20260812 11:18:24.359328 18492 tablet_replica.cc:330] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: stopping tablet replica
I20260812 11:18:24.360026 18492 raft_consensus.cc:2229] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:18:24.360651 18492 raft_consensus.cc:2258] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:24.363044 18492 ts_tablet_manager.cc:1905] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:24.366585 18492 ts_tablet_manager.cc:1918] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:24.366811 18492 log.cc:1196] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/803c8d29e95740b9b9930aa703225909
I20260812 11:18:24.367276 18492 ts_tablet_manager.cc:1939] T 803c8d29e95740b9b9930aa703225909 P e5131b92bf6545079aff20123940b83d: Deleting consensus metadata
W20260812 11:18:24.369130 17590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:34219 (error 108)
I20260812 11:18:24.369171 17592 catalog_manager.cc:4841] TS e5131b92bf6545079aff20123940b83d (127.27.109.3:44629): tablet 803c8d29e95740b9b9930aa703225909 (table test-table [id=f9f5273de861492894b42230e7257574]) successfully deleted
I20260812 11:18:24.369571 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 17959
I20260812 11:18:24.399536 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.370357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60814)
0812 11:18:24.399227 (+ 28870us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.399378 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28687}
W20260812 11:18:24.409704 17590 catalog_manager.cc:4584] TS 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 803c8d29e95740b9b9930aa703225909: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
I20260812 11:18:24.459220 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 18187
I20260812 11:18:24.516983 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.516132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60824)
0812 11:18:24.516644 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.516726 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260812 11:18:24.518400 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:24.518045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60840)
0812 11:18:24.518203 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:24.518299 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 11:18:24.527244 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 17556
I20260812 11:18:24.549853 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46227
--webserver_interface=127.27.109.62
--webserver_port=45709
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:24.702332 18494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:24.702677 18494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:24.702720 18494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:24.707229 18494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:24.707327 18494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:24.707348 18494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:24.707366 18494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:24.707382 18494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:24.713833 18494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=45709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:24.715201 18494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:24.716357 18494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:24.736945 18507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:24.737231 18505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:24.737295 18494 server_base.cc:1034] running on GCE node
W20260812 11:18:24.737479 18511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:24.738390 18494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:24.740028 18494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:24.741405 18494 hybrid_clock.cc:648] HybridClock initialized: now 1786533504741367 us; error 64 us; skew 500 ppm
Aug 12 11:18:24 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:24.745757 18494 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:24.747232 18494 webserver.cc:460] Webserver started at http://127.27.109.62:45709/ using document root <none> and password file <none>
I20260812 11:18:24.748021 18494 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:24.748175 18494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:24.753790 18494 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:18:24.757566 18527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:24.759994 18494 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:18:24.760255 18494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "32cf5ccd919648799de1c6dafc954386"
format_stamp: "Formatted at 2026-08-12 11:18:22 on dist-test-slave-7z32"
server_key: "aabc6bf5ac61e04a2b22a33557de4a24"
server_key_iv: "b87cc90ceccb195e2bf7d1aa5b28998b"
server_key_version: "encryptionkey@0"
I20260812 11:18:24.761183 18494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:24.780797 18494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:24.785149 18494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:24.785442 18494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:24.796066 18494 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:46227
I20260812 11:18:24.796669 18622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:46227 every 8 connection(s)
I20260812 11:18:24.797544 18494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:24.803717 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 18494
I20260812 11:18:24.804275 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:34219
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:46227
--webserver_port=33511
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:24.804750 18623 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:18:24.807397 18623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Bootstrap starting.
I20260812 11:18:24.839778 18623 log.cc:824] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:24.856185 18623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:18:24.857121 18623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Bootstrap complete.
I20260812 11:18:24.863539 18623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:24.864135 18623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cf5ccd919648799de1c6dafc954386, State: Initialized, Role: FOLLOWER
I20260812 11:18:24.864940 18623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:24.865123 18623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:18:24.865227 18623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:18:24.865370 18623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:18:24.868149 18623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:24.868615 18623 leader_election.cc:302] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32cf5ccd919648799de1c6dafc954386; no voters:
I20260812 11:18:24.869115 18623 leader_election.cc:288] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:18:24.869223 18630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:18:24.869477 18630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 LEADER]: Becoming Leader. State: Replica: 32cf5ccd919648799de1c6dafc954386, State: Running, Role: LEADER
I20260812 11:18:24.869864 18630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:24.870684 18623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:24.871119 18636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32cf5ccd919648799de1c6dafc954386. Latest consensus state: current_term: 2 leader_uuid: "32cf5ccd919648799de1c6dafc954386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } } }
I20260812 11:18:24.871364 18636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:24.871661 18634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "32cf5ccd919648799de1c6dafc954386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } } }
I20260812 11:18:24.871815 18634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:24.872470 18638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:24.879638 18638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:24.880596 18638 catalog_manager.cc:1259] Loaded cluster ID: cfe457661ebe4a4aa2388bd1663406e7
I20260812 11:18:24.880682 18638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:18:24.882211 18638 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 11:18:24.885562 18638 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 11:18:24.892112 18638 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:18:24.893074 18638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Loaded TSK: 0
I20260812 11:18:24.894328 18638 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:18:24.997221 18626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:24.998596 18626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:24.998667 18626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:25.006193 18626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:25.006608 18626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:25.006707 18626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:18:25.016796 18626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:34219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=33511
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:25.018883 18626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:25.022583 18626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:25.040225 18666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:25.040225 18667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:25.040421 18669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:25.041249 18626 server_base.cc:1034] running on GCE node
I20260812 11:18:25.041867 18626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:25.043467 18626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:25.044836 18626 hybrid_clock.cc:648] HybridClock initialized: now 1786533505044732 us; error 139 us; skew 500 ppm
Aug 12 11:18:25 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:25.049989 18626 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:18:25.051587 18626 webserver.cc:460] Webserver started at http://127.27.109.1:33511/ using document root <none> and password file <none>
I20260812 11:18:25.052399 18626 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:25.052568 18626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:25.059465 18626 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:25.063342 18686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:25.065725 18626 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:18:25.065943 18626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "1421b1e4e4c54c3e81ff323d791e0555"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "06d3c7fe16e10389c3ac438cb59191c8"
server_key_iv: "ab7700a66849c721871e9b192b1b9240"
server_key_version: "encryptionkey@0"
I20260812 11:18:25.066655 18626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:25.094170 18626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:25.098694 18626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:25.099053 18626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:25.099984 18626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:25.101742 18696 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:18:25.108126 18626 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:18:25.108394 18626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260812 11:18:25.108525 18626 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:18:25.111902 18626 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:18:25.112010 18626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:25.112684 18696 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap starting.
I20260812 11:18:25.127223 18626 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34219
I20260812 11:18:25.128017 18891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34219 every 8 connection(s)
I20260812 11:18:25.129273 18626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:18:25.130131 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 18626
I20260812 11:18:25.130698 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:42839
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:46227
--webserver_port=43435
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:25.140851 18696 log.cc:824] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:18:25 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:25.154335 18696 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:18:25.155040 18696 tablet_bootstrap.cc:492] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap complete.
I20260812 11:18:25.156684 18696 ts_tablet_manager.cc:1397] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent bootstrapping tablet: real 0.044s user 0.033s sys 0.009s
I20260812 11:18:25.159369 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.133006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:37229)
0812 11:18:25.134685 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.134762 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.136150 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.138050 (+ 1900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.138065 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.138926 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:25.140033 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.140071 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.142773 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.142781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.144504 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.144525 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.146191 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.146196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.146456 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.147496 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:25.147525 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:25.150245 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:25.154393 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.154405 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.154432 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.155014 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.155667 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.155675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.155680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.155937 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.156967 (+ 1030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:25.156976 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:25.157373 (+ 397us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.157614 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.158071 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":205,"threads_started":1}
I20260812 11:18:25.160200 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.133161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:37229 (local address 127.27.109.62:46227)
0812 11:18:25.134475 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.134505 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.136526 (+ 2021us) server_negotiation.cc:407] Connection header received
0812 11:18:25.136859 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.136876 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.137315 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.137705 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:25.140461 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.142406 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.144777 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.145896 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.146193 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.151032 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:25.151043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:25.151069 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:25.151130 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:25.154129 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.155237 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.155246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.155253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.155504 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.156107 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.156111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.156113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.156727 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:25.156886 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.157925 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.158426 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":332,"threads_started":1}
I20260812 11:18:25.160595 18696 raft_consensus.cc:348] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.161252 18696 raft_consensus.cc:729] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1421b1e4e4c54c3e81ff323d791e0555, State: Initialized, Role: FOLLOWER
I20260812 11:18:25.161897 18696 consensus_queue.cc:260] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.163311 18696 ts_tablet_manager.cc:1428] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 11:18:25.163398 18892 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46227
I20260812 11:18:25.163686 18892 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:25.164389 18892 heartbeater.cc:511] Master 127.27.109.62:46227 requested a full tablet report, sending...
I20260812 11:18:25.167949 18547 ts_manager.cc:194] Registered new tserver with Master: 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219)
I20260812 11:18:25.172748 18547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:37229
I20260812 11:18:25.176620 18892 heartbeater.cc:503] Master 127.27.109.62:46227 was elected leader, sending a full tablet report...
Aug 12 11:18:25 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:18:25.186652 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.172038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60856)
0812 11:18:25.172932 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.172993 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.173205 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.173977 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.173986 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.174585 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:25.174722 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.174730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.176086 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.176093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.176838 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.176853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.177662 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.177666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.177794 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.178543 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:25.178557 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:25.181123 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:25.184101 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.184124 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.184134 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.184497 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.184902 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.184906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.184908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.185011 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.185672 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:25.185722 (+ 50us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:25.185897 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.186085 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.186424 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":159,"threads_started":1}
I20260812 11:18:25.186652 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.172177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.27.109.1:34219)
0812 11:18:25.173375 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.173390 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.173425 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:18:25.173525 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.173531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.173753 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.173966 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:25.174935 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.175940 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.176998 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.177512 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.177649 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.181419 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:25.181435 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:25.181455 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:25.181492 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:25.183878 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.184673 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.184680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.184681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.184783 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.185140 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.185143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.185147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.185517 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:25.185740 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.186190 (+ 450us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.186422 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":1031,"thread_start_us":166,"threads_started":1}
I20260812 11:18:25.188309 18774 tablet_service.cc:1514] Processing DeleteTablet for tablet 803c8d29e95740b9b9930aa703225909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:24 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:60856
I20260812 11:18:25.190305 18905 tablet_replica.cc:330] stopping tablet replica
I20260812 11:18:25.190982 18905 raft_consensus.cc:2229] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:18:25.191395 18905 raft_consensus.cc:2258] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:25.192919 18905 ts_tablet_manager.cc:1905] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:25.199939 18905 ts_tablet_manager.cc:1918] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:25.200529 18905 log.cc:1196] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/803c8d29e95740b9b9930aa703225909
I20260812 11:18:25.201270 18905 ts_tablet_manager.cc:1939] T 803c8d29e95740b9b9930aa703225909 P 1421b1e4e4c54c3e81ff323d791e0555: Deleting consensus metadata
I20260812 11:18:25.204358 18531 catalog_manager.cc:4841] TS 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219): tablet 803c8d29e95740b9b9930aa703225909 (table test-table [id=f9f5273de861492894b42230e7257574]) successfully deleted
W20260812 11:18:25.284096 18896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:25.284518 18896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:25.284610 18896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:25.289232 18896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:25.289371 18896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:25.289590 18896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:18:25.295962 18896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:42839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=43435
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:25.297427 18896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:25.298729 18896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:25.311626 18923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:25.311653 18918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:25.311671 18913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:25.312258 18896 server_base.cc:1034] running on GCE node
I20260812 11:18:25.312908 18896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:25.313908 18896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:25.315150 18896 hybrid_clock.cc:648] HybridClock initialized: now 1786533505315128 us; error 50 us; skew 500 ppm
Aug 12 11:18:25 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:25.319423 18896 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:18:25.321376 18896 webserver.cc:460] Webserver started at http://127.27.109.2:43435/ using document root <none> and password file <none>
I20260812 11:18:25.322427 18896 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:25.322568 18896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:25.329034 18896 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 11:18:25.333115 18935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:25.335572 18896 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:18:25.335862 18896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "4ec79a765d4d4a16b3ec7be780fec4e5"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "dac6c04f14125a00a44fd6b957a35d16"
server_key_iv: "0ce1951231a210ada0213e6faedd4660"
server_key_version: "encryptionkey@0"
I20260812 11:18:25.336825 18896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:25.358372 18896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:25.364580 18896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:25.364951 18896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:25.366091 18896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:25.367456 18896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:25.367528 18896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:25.367585 18896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:25.367661 18896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:25.384969 18896 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:42839
I20260812 11:18:25.384991 19130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:42839 every 8 connection(s)
I20260812 11:18:25.386653 18896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:18:25.395839 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 18896
I20260812 11:18:25.396390 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:44629
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:46227
--webserver_port=43673
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:18:25 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:25.408697 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.389758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:60997 (local address 127.27.109.62:46227)
0812 11:18:25.389960 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.389965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.392033 (+ 2068us) server_negotiation.cc:407] Connection header received
0812 11:18:25.393277 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.393281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.393358 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.393493 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:25.395479 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.396227 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.398068 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.398435 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.398613 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.403055 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:25.403061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:25.403063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:25.403107 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:25.405452 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.406400 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.406406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.406409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.406512 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.407102 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.407106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.407108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.407376 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:25.407515 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.408265 (+ 750us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.408484 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:18:25.409934 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.390189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:60997)
0812 11:18:25.391728 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.391803 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.392946 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.393817 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.393827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.394375 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:25.395202 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.395223 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.396401 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.396410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.397827 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.397846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.398731 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.398737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.399115 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.400358 (+ 1243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:25.400384 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:25.402629 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:25.405767 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.405787 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.405812 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.406240 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.406639 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.406643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.406645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.406943 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.407516 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:25.407529 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:25.407929 (+ 400us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.408190 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.408686 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"thread_start_us":160,"threads_started":1}
I20260812 11:18:25.412014 19131 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46227
I20260812 11:18:25.412437 19131 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:25.413293 19131 heartbeater.cc:511] Master 127.27.109.62:46227 requested a full tablet report, sending...
I20260812 11:18:25.415515 18547 ts_manager.cc:194] Registered new tserver with Master: 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839)
I20260812 11:18:25.416441 18547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:60997
W20260812 11:18:25.559201 19140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:25.559640 19140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:25.559703 19140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:25.564790 19140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:25.564914 19140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:25.564991 19140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:18:25.572404 19140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:44629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=43673
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:25.574334 19140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:25.576083 19140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:25.593158 19154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:25.593199 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:25.593199 19149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:25.593865 19140 server_base.cc:1034] running on GCE node
I20260812 11:18:25.594599 19140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:25.595679 19140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:25.599114 19140 hybrid_clock.cc:648] HybridClock initialized: now 1786533505598954 us; error 164 us; skew 500 ppm
Aug 12 11:18:25 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:25.604446 19140 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:18:25.606349 19140 webserver.cc:460] Webserver started at http://127.27.109.3:43673/ using document root <none> and password file <none>
I20260812 11:18:25.607512 19140 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:25.607748 19140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:25.614917 19140 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:18:25.619041 19165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:25.622087 19140 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:18:25.622733 19140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "e5131b92bf6545079aff20123940b83d"
format_stamp: "Formatted at 2026-08-12 11:18:24 on dist-test-slave-7z32"
server_key: "a0a0b951d55e6fe433de73ee748f5781"
server_key_iv: "f3ed40abbcfb0375882a3709947ae503"
server_key_version: "encryptionkey@0"
I20260812 11:18:25.624241 19140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:25.649290 19140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:25.655063 19140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:25.655386 19140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:25.656320 19140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:25.657641 19140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:25.657742 19140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:25.657838 19140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:25.657894 19140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:25.675740 19140 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:44629
I20260812 11:18:25.675856 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:44629 every 8 connection(s)
I20260812 11:18:25.677400 19140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:18:25.680023 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 19140
I20260812 11:18:25.696987 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.680905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.27.109.62:46227)
0812 11:18:25.681060 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.681063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.681675 (+ 612us) server_negotiation.cc:407] Connection header received
0812 11:18:25.681884 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.681887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.681949 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.682048 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:25.683609 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.684351 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.686578 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.687063 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.687226 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.689808 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:25.689813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:25.689816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:25.689861 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:25.692613 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.693896 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.693902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.693904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.694254 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.695032 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.695036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.695038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.695565 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:25.695814 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.696392 (+ 578us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.696638 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 11:18:25 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:25.707938 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.680905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39465 (local address 127.27.109.62:46227)
0812 11:18:25.681550 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.681556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.682798 (+ 1242us) server_negotiation.cc:407] Connection header received
0812 11:18:25.686253 (+ 3455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.686258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.686348 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.686518 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:25.689001 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.690076 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.693018 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.693508 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.693663 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.701274 (+ 7611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:25.701278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:25.701281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:25.701385 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:25.703423 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.704577 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.704581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.704582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.704703 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:25.705592 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:25.705596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:25.705597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:25.705875 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:25.706007 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.707148 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.707726 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":524,"thread_start_us":161,"threads_started":1}
I20260812 11:18:25.708863 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.681669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:39465)
0812 11:18:25.682434 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.682510 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.685532 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.687057 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.687074 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.687865 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:25.688724 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.688752 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.690352 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.690356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.692633 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.692654 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.694156 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.694173 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.694808 (+ 635us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.696424 (+ 1616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:25.696460 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:25.700875 (+ 4415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:25.703726 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.703738 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.703767 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.704329 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.705016 (+ 687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:25.705021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:25.705024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:25.705311 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:25.706054 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:25.706063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:25.706539 (+ 476us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.706806 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.707450 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":201,"threads_started":1}
I20260812 11:18:25.711815 19362 heartbeater.cc:347] Connected to a master server at 127.27.109.62:46227
I20260812 11:18:25.712319 19362 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:25.713878 19362 heartbeater.cc:511] Master 127.27.109.62:46227 requested a full tablet report, sending...
I20260812 11:18:25.716511 18545 ts_manager.cc:194] Registered new tserver with Master: e5131b92bf6545079aff20123940b83d (127.27.109.3:44629)
I20260812 11:18:25.717626 18545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:39465
I20260812 11:18:25.732053 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:18:25.741403 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.733815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.27.109.62:46227)
0812 11:18:25.734402 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.734407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.734433 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:18:25.734572 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.734575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.734723 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.734945 (+ 222us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:18:25.735497 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.736558 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.738572 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.739021 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.739167 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.739920 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:18:25.740526 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:18:25.740679 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.741023 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.741184 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 11:18:25.743155 18545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:18:25.750082 18545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:18:25.764802 18774 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2f4f95117645ceaa80d7b463093347 (DEFAULT_TABLE table=test-table [id=fd53bcdfedbf43a298a3ba272eb739cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:25.765885 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2f4f95117645ceaa80d7b463093347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:25.770115 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.763388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.27.109.2:42839)
0812 11:18:25.764337 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.764352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.764386 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:18:25.764502 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.764509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.764824 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.765050 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:25.765512 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.766741 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.768303 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.769386 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.769525 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.769716 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.769849 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.769954 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":207,"threads_started":1}
I20260812 11:18:25.770102 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.762944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:42839 (local address 127.0.0.1:55214)
0812 11:18:25.763275 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.763299 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.763446 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.765221 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.765225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.765250 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:25.765392 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.765399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.766964 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.766970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.767946 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.767959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.769531 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.769535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.769647 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.769714 (+ 67us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.769808 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.769929 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":117,"threads_started":1}
I20260812 11:18:25.771709 19377 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap starting.
I20260812 11:18:25.771991 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.763429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:44629 (local address 127.0.0.1:58056)
0812 11:18:25.764055 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.764123 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.764261 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.766485 (+ 2224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.766489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.766513 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:25.766617 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.766623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.768477 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.768482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.769769 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.769788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.771403 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.771409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.771554 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.771604 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.771665 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.771777 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":175,"threads_started":1}
I20260812 11:18:25.773061 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.764224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.27.109.3:44629)
0812 11:18:25.764863 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.764882 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.764916 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:18:25.765022 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.765029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.766232 (+ 1203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.766502 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:25.766770 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.768218 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.770002 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.771198 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.772082 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.772329 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.772464 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.772858 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":241,"threads_started":1}
I20260812 11:18:25.773564 19377 tablet_bootstrap.cc:654] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:25.774474 19012 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2f4f95117645ceaa80d7b463093347 (DEFAULT_TABLE table=test-table [id=fd53bcdfedbf43a298a3ba272eb739cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:25.775667 19377 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: No bootstrap required, opened a new log
I20260812 11:18:25.775885 19377 ts_tablet_manager.cc:1397] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:18:25.775969 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2f4f95117645ceaa80d7b463093347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:25.776499 19377 raft_consensus.cc:348] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.776670 19377 raft_consensus.cc:374] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:25.776737 19377 raft_consensus.cc:729] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1421b1e4e4c54c3e81ff323d791e0555, State: Initialized, Role: FOLLOWER
I20260812 11:18:25.776912 19377 consensus_queue.cc:260] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.777832 19377 ts_tablet_manager.cc:1428] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 11:18:25.777594 19250 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2f4f95117645ceaa80d7b463093347 (DEFAULT_TABLE table=test-table [id=fd53bcdfedbf43a298a3ba272eb739cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:25.778976 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2f4f95117645ceaa80d7b463093347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:25.787014 19382 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Bootstrap starting.
I20260812 11:18:25.789290 19384 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Bootstrap starting.
I20260812 11:18:25.791558 19382 tablet_bootstrap.cc:654] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:25.792847 19382 log.cc:824] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:25.793010 19384 tablet_bootstrap.cc:654] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:25.794266 19384 log.cc:824] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:25.797180 19382 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: No bootstrap required, opened a new log
I20260812 11:18:25.797550 19382 ts_tablet_manager.cc:1397] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260812 11:18:25.797789 19384 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: No bootstrap required, opened a new log
I20260812 11:18:25.798101 19384 ts_tablet_manager.cc:1397] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 11:18:25.802093 19384 raft_consensus.cc:348] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.802521 19384 raft_consensus.cc:374] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:25.802598 19384 raft_consensus.cc:729] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5131b92bf6545079aff20123940b83d, State: Initialized, Role: FOLLOWER
I20260812 11:18:25.803671 19384 consensus_queue.cc:260] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.804679 19362 heartbeater.cc:503] Master 127.27.109.62:46227 was elected leader, sending a full tablet report...
I20260812 11:18:25.805166 19384 ts_tablet_manager.cc:1428] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 11:18:25.805406 19382 raft_consensus.cc:348] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.805856 19382 raft_consensus.cc:374] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:25.805934 19382 raft_consensus.cc:729] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec79a765d4d4a16b3ec7be780fec4e5, State: Initialized, Role: FOLLOWER
I20260812 11:18:25.806144 19390 raft_consensus.cc:482] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:25.806401 19390 raft_consensus.cc:504] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.806658 19382 consensus_queue.cc:260] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.808208 19382 ts_tablet_manager.cc:1428] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260812 11:18:25.808765 19131 heartbeater.cc:503] Master 127.27.109.62:46227 was elected leader, sending a full tablet report...
I20260812 11:18:25.811206 19390 leader_election.cc:288] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839), 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219)
I20260812 11:18:25.815687 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.810068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:56127 (local address 127.27.109.2:42839)
0812 11:18:25.810253 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.810258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.810277 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:18:25.810336 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.810339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.810397 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.810478 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:25.811402 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.812357 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.814297 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.814992 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.815153 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.815214 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.815479 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.815570 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:18:25.816016 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.809041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:42839 (local address 127.27.109.3:56127)
0812 11:18:25.809294 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.809321 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.809479 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.810760 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.810764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.810796 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:25.811016 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.811024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.812586 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.812591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.814014 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.814034 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.815137 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.815141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.815268 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.815348 (+ 80us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.815462 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.815874 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":313}
I20260812 11:18:25.816857 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2f4f95117645ceaa80d7b463093347" candidate_uuid: "e5131b92bf6545079aff20123940b83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" is_pre_election: true
I20260812 11:18:25.817277 19050 raft_consensus.cc:2454] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5131b92bf6545079aff20123940b83d in term 0.
I20260812 11:18:25.817857 19167 leader_election.cc:302] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec79a765d4d4a16b3ec7be780fec4e5, e5131b92bf6545079aff20123940b83d; no voters:
I20260812 11:18:25.818249 19390 raft_consensus.cc:2781] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:18:25.818372 19390 raft_consensus.cc:482] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:18:25.818435 19390 raft_consensus.cc:3037] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:25.819446 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.809539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.27.109.3:37329)
0812 11:18:25.809889 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.809906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:25.810041 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:25.812151 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:25.812155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:25.812178 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:25.812335 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.812342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.815272 (+ 2930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.815277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.817013 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:25.817033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.818601 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.818606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:25.818920 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.818959 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:18:25.819085 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 11:18:25.819227 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":177,"threads_started":1}
I20260812 11:18:25.819516 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.810367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37329 (local address 127.27.109.1:34219)
0812 11:18:25.811593 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.811614 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.811633 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:18:25.811769 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.811773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.811861 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.812005 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:25.812485 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.815078 (+ 2593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.817261 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.818074 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.818265 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.818480 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.819176 (+ 696us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.819344 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":166,"threads_started":1}
I20260812 11:18:25.820176 19390 raft_consensus.cc:504] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.820701 19390 leader_election.cc:288] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [CANDIDATE]: Term 1 election: Requested vote from peers 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839), 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219)
I20260812 11:18:25.820727 18810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2f4f95117645ceaa80d7b463093347" candidate_uuid: "e5131b92bf6545079aff20123940b83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1421b1e4e4c54c3e81ff323d791e0555" is_pre_election: true
I20260812 11:18:25.821216 18810 raft_consensus.cc:2454] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5131b92bf6545079aff20123940b83d in term 0.
I20260812 11:18:25.821374 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2f4f95117645ceaa80d7b463093347" candidate_uuid: "e5131b92bf6545079aff20123940b83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5"
I20260812 11:18:25.821573 19050 raft_consensus.cc:3037] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:25.821667 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2f4f95117645ceaa80d7b463093347" candidate_uuid: "e5131b92bf6545079aff20123940b83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1421b1e4e4c54c3e81ff323d791e0555"
I20260812 11:18:25.821825 18808 raft_consensus.cc:3037] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:25.823347 19050 raft_consensus.cc:2454] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5131b92bf6545079aff20123940b83d in term 1.
I20260812 11:18:25.823653 18808 raft_consensus.cc:2454] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5131b92bf6545079aff20123940b83d in term 1.
I20260812 11:18:25.823734 19167 leader_election.cc:302] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec79a765d4d4a16b3ec7be780fec4e5, e5131b92bf6545079aff20123940b83d; no voters:
I20260812 11:18:25.824035 19390 raft_consensus.cc:2781] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:25.824235 19390 raft_consensus.cc:686] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 1 LEADER]: Becoming Leader. State: Replica: e5131b92bf6545079aff20123940b83d, State: Running, Role: LEADER
I20260812 11:18:25.824717 19390 consensus_queue.cc:237] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:25.829649 18545 catalog_manager.cc:5495] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5131b92bf6545079aff20123940b83d (127.27.109.3). New cstate: current_term: 1 leader_uuid: "e5131b92bf6545079aff20123940b83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } health_report { overall_health: UNKNOWN } } }
I20260812 11:18:25.852188 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.847614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.27.109.3:44629)
0812 11:18:25.847932 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:25.847936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:25.847951 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:18:25.848063 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:25.848066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:25.848148 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:25.848297 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:18:25.848598 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.849186 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.850145 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:25.850379 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:25.850502 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:25.850983 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:18:25.851647 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:18:25.851824 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:25.851872 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 11:18:25.851966 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260812 11:18:25.861426 18808 raft_consensus.cc:1261] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Refusing update from remote peer e5131b92bf6545079aff20123940b83d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:25.861492 19050 raft_consensus.cc:1261] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 FOLLOWER]: Refusing update from remote peer e5131b92bf6545079aff20123940b83d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:25.862738 19393 consensus_queue.cc:1035] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:25.863250 19390 consensus_queue.cc:1035] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:25.875874 19407 mvcc.cc:205] Tried to move back new op lower bound from 7317641239994597376 to 7317641239861882880. Current Snapshot: MvccSnapshot[applied={T|T < 7317641239994597376}]
I20260812 11:18:25.887859 18550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:25.888131 18550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:25.892601 18550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Sending DeleteTablet for 3 replicas of tablet 7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:25.894145 18774 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e2f4f95117645ceaa80d7b463093347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:25 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:60856
I20260812 11:18:25.894279 19012 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e2f4f95117645ceaa80d7b463093347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:25 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:55214
I20260812 11:18:25.894276 19250 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e2f4f95117645ceaa80d7b463093347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:25 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:58056
I20260812 11:18:25.895061 19429 tablet_replica.cc:330] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: stopping tablet replica
I20260812 11:18:25.895350 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 18626
I20260812 11:18:25.953927 19429 raft_consensus.cc:2229] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:18:25.954672 19429 raft_consensus.cc:2258] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:25.960953 19431 tablet_replica.cc:330] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: stopping tablet replica
I20260812 11:18:25.961757 19431 raft_consensus.cc:2229] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:18:25.962487 19431 raft_consensus.cc:2258] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:25.962481 19429 ts_tablet_manager.cc:1905] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:25.967419 19431 ts_tablet_manager.cc:1905] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:25.969439 19429 ts_tablet_manager.cc:1918] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:25.969614 19429 log.cc:1196] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:25.970022 19429 ts_tablet_manager.cc:1939] T 7e2f4f95117645ceaa80d7b463093347 P e5131b92bf6545079aff20123940b83d: Deleting consensus metadata
I20260812 11:18:25.971760 18534 catalog_manager.cc:4841] TS e5131b92bf6545079aff20123940b83d (127.27.109.3:44629): tablet 7e2f4f95117645ceaa80d7b463093347 (table test-table [id=fd53bcdfedbf43a298a3ba272eb739cb]) successfully deleted
I20260812 11:18:25.972913 19431 ts_tablet_manager.cc:1918] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:25.973201 19431 log.cc:1196] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:25.973667 19431 ts_tablet_manager.cc:1939] T 7e2f4f95117645ceaa80d7b463093347 P 4ec79a765d4d4a16b3ec7be780fec4e5: Deleting consensus metadata
I20260812 11:18:25.976198 18531 catalog_manager.cc:4841] TS 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839): tablet 7e2f4f95117645ceaa80d7b463093347 (table test-table [id=fd53bcdfedbf43a298a3ba272eb739cb]) successfully deleted
W20260812 11:18:25.976840 18531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:34219 (error 108)
I20260812 11:18:25.977146 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 18896
I20260812 11:18:25.981067 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:25.978469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60872)
0812 11:18:25.979814 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:25.980446 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1200}
W20260812 11:18:25.981539 18531 catalog_manager.cc:4584] TS 1421b1e4e4c54c3e81ff323d791e0555 (127.27.109.1:34219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e2f4f95117645ceaa80d7b463093347: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
I20260812 11:18:26.016809 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 19140
I20260812 11:18:26.060822 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:26.060399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60878)
0812 11:18:26.060600 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:26.060698 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 11:18:26.063622 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:26.062557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34219 (local address 127.0.0.1:60882)
0812 11:18:26.063406 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:26.063495 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":708}
I20260812 11:18:26.069381 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 18494
I20260812 11:18:26.107985 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46227
--webserver_interface=127.27.109.62
--webserver_port=45709
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:26.277591 19433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:26.278053 19433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:26.278151 19433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:26.284888 19433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:26.285068 19433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:26.285106 19433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:26.285152 19433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:26.285189 19433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:26.292168 19433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=45709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:26.293529 19433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:26.295131 19433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:26.307802 19449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:26.307709 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:26.307911 19445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:26.308562 19433 server_base.cc:1034] running on GCE node
I20260812 11:18:26.309494 19433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:26.311808 19433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:26.315538 19433 hybrid_clock.cc:648] HybridClock initialized: now 1786533506315357 us; error 195 us; skew 500 ppm
Aug 12 11:18:26 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533506, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:26.327030 19433 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:26.329411 19433 webserver.cc:460] Webserver started at http://127.27.109.62:45709/ using document root <none> and password file <none>
I20260812 11:18:26.331032 19433 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:26.331622 19433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:26.341384 19433 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 11:18:26.346004 19456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:26.349116 19433 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260812 11:18:26.349635 19433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "32cf5ccd919648799de1c6dafc954386"
format_stamp: "Formatted at 2026-08-12 11:18:22 on dist-test-slave-7z32"
server_key: "aabc6bf5ac61e04a2b22a33557de4a24"
server_key_iv: "b87cc90ceccb195e2bf7d1aa5b28998b"
server_key_version: "encryptionkey@0"
I20260812 11:18:26.350760 19433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:26.375066 19433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:26.380401 19433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:26.380736 19433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:26.392542 19433 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:46227
I20260812 11:18:26.393056 19547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:46227 every 8 connection(s)
I20260812 11:18:26.394330 19433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:26.394608 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 19433
I20260812 11:18:26.395318 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:34219
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:46227
--webserver_port=33511
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:26.402225 19548 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:18:26.405884 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Bootstrap starting.
I20260812 11:18:26.434657 19548 log.cc:824] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:26.452488 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:18:26.453100 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: Bootstrap complete.
I20260812 11:18:26.460247 19548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:26.461082 19548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cf5ccd919648799de1c6dafc954386, State: Initialized, Role: FOLLOWER
I20260812 11:18:26.461756 19548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:26.461953 19548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:18:26.462033 19548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:18:26.462139 19548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:18:26.465691 19548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:26.466218 19548 leader_election.cc:302] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32cf5ccd919648799de1c6dafc954386; no voters:
I20260812 11:18:26.466729 19548 leader_election.cc:288] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 11:18:26.467264 19553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 11:18:26.467710 19553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [term 3 LEADER]: Becoming Leader. State: Replica: 32cf5ccd919648799de1c6dafc954386, State: Running, Role: LEADER
I20260812 11:18:26.468241 19553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } }
I20260812 11:18:26.468689 19548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:26.470564 19561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32cf5ccd919648799de1c6dafc954386. Latest consensus state: current_term: 3 leader_uuid: "32cf5ccd919648799de1c6dafc954386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } } }
I20260812 11:18:26.470849 19561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:26.470636 19557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "32cf5ccd919648799de1c6dafc954386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cf5ccd919648799de1c6dafc954386" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46227 } } }
I20260812 11:18:26.471236 19557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386 [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:26.476034 19580 catalog_manager.cc:1259] Loaded cluster ID: cfe457661ebe4a4aa2388bd1663406e7
I20260812 11:18:26.476167 19580 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: loading cluster ID for follower catalog manager: success
W20260812 11:18:26.477741 19580 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 32cf5ccd919648799de1c6dafc954386: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260812 11:18:26.477870 19580 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260812 11:18:26.478025 19578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:26.483455 19578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:26.484436 19578 catalog_manager.cc:1259] Loaded cluster ID: cfe457661ebe4a4aa2388bd1663406e7
I20260812 11:18:26.484531 19578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 11:18:26.485473 19578 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0375e139ad google::LogMessage::Fail() at ??:0
@ 0x7f0375e1688a google::LogMessage::SendToLog() at ??:0
@ 0x7f0375e134de google::LogMessage::Flush() at ??:0
@ 0x7f0375e14a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0380752fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0380753d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0380750579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0380794640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f03780653d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0378065c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f03780673f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0378056512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f037969a6db start_thread at ??:0
@ 0x7f037491161f clone at ??:0
W20260812 11:18:26.623652 19551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:26.624127 19551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:26.624193 19551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:26.629882 19551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:26.630034 19551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:26.630121 19551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:18:26.638041 19551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:34219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=33511
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:26.639993 19551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:26.641690 19551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:26.658569 19614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:26.658569 19612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:26.658981 19616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:26.660324 19551 server_base.cc:1034] running on GCE node
I20260812 11:18:26.661123 19551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:26.662483 19551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:26.665984 19551 hybrid_clock.cc:648] HybridClock initialized: now 1786533506665894 us; error 105 us; skew 500 ppm
Aug 12 11:18:26 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533506, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:26.672118 19551 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:18:26.674057 19551 webserver.cc:460] Webserver started at http://127.27.109.1:33511/ using document root <none> and password file <none>
I20260812 11:18:26.675487 19551 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:26.675634 19551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:26.683698 19551 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:18:26.689082 19626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:26.692479 19551 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260812 11:18:26.692725 19551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "1421b1e4e4c54c3e81ff323d791e0555"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "06d3c7fe16e10389c3ac438cb59191c8"
server_key_iv: "ab7700a66849c721871e9b192b1b9240"
server_key_version: "encryptionkey@0"
I20260812 11:18:26.693796 19551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:26.736418 19551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:26.741217 19551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:26.741547 19551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:26.742406 19551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:26.744186 19637 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:18:26.754150 19551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:18:26.754407 19551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20260812 11:18:26.754572 19551 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:18:26.757192 19551 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:18:26.757329 19551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260812 11:18:26.757932 19637 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap starting.
I20260812 11:18:26.774003 19551 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34219
I20260812 11:18:26.774447 19816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34219 every 8 connection(s)
I20260812 11:18:26.776036 19551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:18:26.782655 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 19551
I20260812 11:18:26.783396 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:42839
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:46227
--webserver_port=43435
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:26.804170 19637 log.cc:824] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:26.814800 19637 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:18:26.815454 19637 tablet_bootstrap.cc:492] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Bootstrap complete.
I20260812 11:18:26.816920 19637 ts_tablet_manager.cc:1397] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent bootstrapping tablet: real 0.059s user 0.029s sys 0.024s
I20260812 11:18:26.820411 19637 raft_consensus.cc:348] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:26.821014 19637 raft_consensus.cc:729] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1421b1e4e4c54c3e81ff323d791e0555, State: Initialized, Role: FOLLOWER
I20260812 11:18:26.821688 19637 consensus_queue.cc:260] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:26.823047 19637 ts_tablet_manager.cc:1428] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260812 11:18:26.857173 19819 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:18:26.779067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:39073)
0812 11:18:26.779743 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:26.779816 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:26.781317 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:26.856165 (+ 74848us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":218,"threads_started":1}
W20260812 11:18:26.858126 19817 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:46227 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:46227: Client connection negotiation failed: client connection to 127.27.109.62:46227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:18:26.859462 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:26.859016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:41795)
0812 11:18:26.859262 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:26.859372 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260812 11:18:26.860458 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:26.860113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:42717)
0812 11:18:26.860256 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:26.860394 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 11:18:26.860694 19817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:18:27.007213 19821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:27.007767 19821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:27.007872 19821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:27.014966 19821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:27.015331 19821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:27.016384 19821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:18:27.030961 19821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:42839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=43435
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:27.032938 19821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:27.035115 19821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:27.053952 19828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:27.053917 19832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:27.054432 19821 server_base.cc:1034] running on GCE node
W20260812 11:18:27.053917 19829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:27.055090 19821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:27.055997 19821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:27.057328 19821 hybrid_clock.cc:648] HybridClock initialized: now 1786533507057260 us; error 103 us; skew 500 ppm
Aug 12 11:18:27 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:27.061632 19821 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:18:27.062912 19821 webserver.cc:460] Webserver started at http://127.27.109.2:43435/ using document root <none> and password file <none>
I20260812 11:18:27.063719 19821 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:27.063856 19821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:27.069814 19821 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:18:27.073422 19845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:27.076018 19821 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:18:27.076370 19821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "4ec79a765d4d4a16b3ec7be780fec4e5"
format_stamp: "Formatted at 2026-08-12 11:18:23 on dist-test-slave-7z32"
server_key: "dac6c04f14125a00a44fd6b957a35d16"
server_key_iv: "0ce1951231a210ada0213e6faedd4660"
server_key_version: "encryptionkey@0"
I20260812 11:18:27.077242 19821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:27.099478 19821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:27.104593 19821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:27.104913 19821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:27.105862 19821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:27.107239 19821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:27.107306 19821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:27.107359 19821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:27.107376 19821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:27.121508 19821 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:42839
I20260812 11:18:27.121600 20049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:42839 every 8 connection(s)
I20260812 11:18:27.123261 19821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:18:27.125968 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 19821
I20260812 11:18:27.126516 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:44629
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:46227
--webserver_port=43673
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:18:27.129472 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:27.126813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:35021)
0812 11:18:27.127759 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:27.128461 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":201,"threads_started":1}
W20260812 11:18:27.130339 20051 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:46227 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:46227: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
I20260812 11:18:27.131412 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:27.131138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:32801)
0812 11:18:27.131274 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:27.131333 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 11:18:27.132490 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:27.132193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:43993)
0812 11:18:27.132378 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:27.132435 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260812 11:18:27.132922 20051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:18:27.293617 20058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:27.294066 20058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:27.294121 20058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:27.299098 20058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:27.299211 20058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:27.299286 20058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:18:27.306371 20058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:44629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=43673
--tserver_master_addrs=127.27.109.62:46227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:27.308243 20058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:27.309712 20058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:27.324559 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:27.324560 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:27.324560 20072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:27.325143 20058 server_base.cc:1034] running on GCE node
I20260812 11:18:27.325995 20058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:27.327106 20058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:27.328327 20058 hybrid_clock.cc:648] HybridClock initialized: now 1786533507328303 us; error 47 us; skew 500 ppm
Aug 12 11:18:27 dist-test-slave-7z32 krb5kdc[17536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:27.333127 20058 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:18:27.335100 20058 webserver.cc:460] Webserver started at http://127.27.109.3:43673/ using document root <none> and password file <none>
I20260812 11:18:27.335994 20058 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:27.336093 20058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:27.342612 20058 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 11:18:27.346436 20086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:27.348869 20058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:27.349156 20058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "e5131b92bf6545079aff20123940b83d"
format_stamp: "Formatted at 2026-08-12 11:18:24 on dist-test-slave-7z32"
server_key: "a0a0b951d55e6fe433de73ee748f5781"
server_key_iv: "f3ed40abbcfb0375882a3709947ae503"
server_key_version: "encryptionkey@0"
I20260812 11:18:27.350124 20058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:27.395920 20058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:27.401010 20058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:27.401341 20058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:27.402259 20058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:27.403645 20058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:27.403740 20058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:27.403836 20058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:27.403887 20058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:27.416993 20058 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:44629
I20260812 11:18:27.417172 20271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:44629 every 8 connection(s)
I20260812 11:18:27.418408 20058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:18:27.421262 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 20058
I20260812 11:18:27.424832 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:27.422005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:35897)
0812 11:18:27.423134 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:27.423744 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":292,"threads_started":1}
W20260812 11:18:27.425871 20272 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:46227 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:46227: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
I20260812 11:18:27.427457 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:27.426575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:51251)
0812 11:18:27.427188 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:27.427355 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260812 11:18:27.429133 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:27.428626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:55249)
0812 11:18:27.428856 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:27.429024 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260812 11:18:27.429616 20272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:18:27.862130 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:27.861408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:38811)
0812 11:18:27.861835 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:27.861935 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":97,"threads_started":1}
I20260812 11:18:28.134387 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:28.133649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:58081)
0812 11:18:28.134106 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:28.134207 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":97,"threads_started":1}
I20260812 11:18:28.431609 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:28.430715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:33161)
0812 11:18:28.431344 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:28.431451 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":336,"threads_started":1}
I20260812 11:18:28.568704 20283 raft_consensus.cc:482] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:28.568967 20283 raft_consensus.cc:504] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:28.570355 20283 leader_election.cc:288] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839), e5131b92bf6545079aff20123940b83d (127.27.109.3:44629)
Aug 12 11:18:28 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533506, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
Aug 12 11:18:28 dist-test-slave-7z32 krb5kdc[17536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533506, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260812 11:18:28.590308 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:28.571270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:49467 (local address 127.27.109.2:42839)
0812 11:18:28.571806 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:28.571841 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:28.571904 (+ 63us) server_negotiation.cc:407] Connection header received
0812 11:18:28.572676 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:28.572683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:28.573089 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:28.573401 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:28.575868 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.577705 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.579133 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.579861 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.580109 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:28.584022 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:28.584031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:28.584050 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:28.584099 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:28.586494 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:28.587566 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:28.587574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:28.587580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:28.587811 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:28.588351 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:28.588355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:28.588356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:28.588940 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:28.589136 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:28.589729 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 11:18:28.590057 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":80,"threads_started":1}
I20260812 11:18:28.590301 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:28.571177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43979 (local address 127.27.109.3:44629)
0812 11:18:28.571848 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:28.571891 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:28.571941 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:18:28.572789 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:28.572795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:28.573232 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:28.573637 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:28.575514 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.576953 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.579184 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.579813 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.580043 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:28.584045 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:28.584053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:28.584075 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:28.584120 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:28.586551 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:28.587617 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:28.587630 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:28.587641 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:28.587884 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:28.588321 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:28.588326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:28.588328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:28.588894 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:28.589025 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:28.589734 (+ 709us) server_negotiation.cc:299] Negotiation successful
0812 11:18:28.590057 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":119,"threads_started":1}
I20260812 11:18:28.590289 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:28.570814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:44629 (local address 127.27.109.1:43979)
0812 11:18:28.571550 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:28.571571 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:28.571753 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:28.573618 (+ 1865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:28.573622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:28.574353 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:28.575251 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:28.575272 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.577266 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.577271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:28.578966 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:28.578982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.579991 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.579999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:28.580222 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:28.581139 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:28.581169 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:28.583613 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:28.586853 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:28.586858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:28.586860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:28.587344 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:28.588025 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:28.588028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:28.588031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:28.588171 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:28.589040 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:28.589052 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:28.589349 (+ 297us) client_negotiation.cc:769] Sending connection context
0812 11:18:28.589575 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 11:18:28.590073 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":229,"spinlock_wait_cycles":152064,"thread_start_us":102,"threads_started":1}
I20260812 11:18:28.590507 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:28.570814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:42839 (local address 127.27.109.1:49467)
0812 11:18:28.571426 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:28.571446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:28.571622 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:28.573599 (+ 1977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:28.573607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:28.574350 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:28.575247 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:28.575272 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.577971 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.577975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:28.578940 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:28.578954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:28.580084 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:28.580096 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:28.580308 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:28.581212 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:28.581226 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:28.583663 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:28.586713 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:28.586722 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:28.586741 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:28.587362 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:28.587973 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:28.587977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:28.587981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:28.588183 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:28.589108 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:28.589111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:28.589347 (+ 236us) client_negotiation.cc:769] Sending connection context
0812 11:18:28.589575 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 11:18:28.590184 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":422,"spinlock_wait_cycles":46208,"thread_start_us":167,"threads_started":1}
W20260812 11:18:28.593724 19630 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
W20260812 11:18:28.593994 19632 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5131b92bf6545079aff20123940b83d (127.27.109.3:44629): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:28.594177 19632 leader_election.cc:302] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1421b1e4e4c54c3e81ff323d791e0555; no voters: 4ec79a765d4d4a16b3ec7be780fec4e5, e5131b92bf6545079aff20123940b83d
I20260812 11:18:28.594587 20283 raft_consensus.cc:2726] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:18:28.864078 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:28.863584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:48969)
0812 11:18:28.863837 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:28.863944 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260812 11:18:29.136441 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:29.135771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:54095)
0812 11:18:29.136172 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:29.136263 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":118,"threads_started":1}
I20260812 11:18:29.433763 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:29.433007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:40643)
0812 11:18:29.433450 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:29.433565 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":112,"threads_started":1}
I20260812 11:18:29.866173 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:29.865473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:47043)
0812 11:18:29.865948 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:29.866046 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":129,"threads_started":1}
I20260812 11:18:30.115262 20293 raft_consensus.cc:482] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:30.115442 20293 raft_consensus.cc:504] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:30.116007 20293 leader_election.cc:288] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839), e5131b92bf6545079aff20123940b83d (127.27.109.3:44629)
W20260812 11:18:30.117373 19632 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5131b92bf6545079aff20123940b83d (127.27.109.3:44629): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
W20260812 11:18:30.117666 19630 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:30.117897 19630 leader_election.cc:302] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1421b1e4e4c54c3e81ff323d791e0555; no voters: 4ec79a765d4d4a16b3ec7be780fec4e5, e5131b92bf6545079aff20123940b83d
I20260812 11:18:30.118209 20293 raft_consensus.cc:2726] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:18:30.138278 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:30.137568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:50345)
0812 11:18:30.138023 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:30.138113 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":133,"threads_started":1}
I20260812 11:18:30.435664 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:30.435113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:52397)
0812 11:18:30.435434 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:30.435503 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260812 11:18:30.868083 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:30.867469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:51197)
0812 11:18:30.867841 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:30.867931 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260812 11:18:31.140305 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:31.139458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:39173)
0812 11:18:31.139977 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:31.140145 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":94,"threads_started":1}
I20260812 11:18:31.439432 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:31.438676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:34285)
0812 11:18:31.439196 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:31.439288 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":181,"threads_started":1}
I20260812 11:18:31.870208 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:31.869275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:49791)
0812 11:18:31.869899 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:31.870005 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":109,"threads_started":1}
I20260812 11:18:32.142274 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:32.141651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:52115)
0812 11:18:32.142031 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:32.142134 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
I20260812 11:18:32.441363 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:32.440720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:45967)
0812 11:18:32.441116 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:32.441224 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":104,"threads_started":1}
I20260812 11:18:32.872480 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:32.871699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:59895)
0812 11:18:32.872211 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:32.872317 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":146,"threads_started":1}
I20260812 11:18:33.144361 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:33.143635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:36705)
0812 11:18:33.144133 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:33.144230 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":126,"threads_started":1}
I20260812 11:18:33.263962 20315 raft_consensus.cc:482] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:33.264216 20315 raft_consensus.cc:504] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:33.264788 20315 leader_election.cc:288] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839), e5131b92bf6545079aff20123940b83d (127.27.109.3:44629)
W20260812 11:18:33.266111 19630 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
W20260812 11:18:33.266292 19632 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5131b92bf6545079aff20123940b83d (127.27.109.3:44629): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:33.266376 19632 leader_election.cc:302] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1421b1e4e4c54c3e81ff323d791e0555; no voters: 4ec79a765d4d4a16b3ec7be780fec4e5, e5131b92bf6545079aff20123940b83d
I20260812 11:18:33.266580 20315 raft_consensus.cc:2726] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:18:33.443313 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:33.442553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:49177)
0812 11:18:33.443112 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:33.443188 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":255,"threads_started":1}
I20260812 11:18:33.874316 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:33.873709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:39955)
0812 11:18:33.874118 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:33.874203 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
I20260812 11:18:34.146561 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:34.145785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:56745)
0812 11:18:34.146314 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:34.146413 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":99,"threads_started":1}
I20260812 11:18:34.445641 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:34.444878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:35847)
0812 11:18:34.445436 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:34.445528 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":135,"threads_started":1}
I20260812 11:18:34.876474 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:34.875781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:58937)
0812 11:18:34.876233 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:34.876341 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":153,"threads_started":1}
I20260812 11:18:35.148684 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:35.147955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:41025)
0812 11:18:35.148400 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:35.148511 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260812 11:18:35.447566 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:35.446737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:50949)
0812 11:18:35.447279 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:35.447395 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":187,"threads_started":1}
I20260812 11:18:35.878607 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:35.877916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:39279)
0812 11:18:35.878348 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:35.878463 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":101,"threads_started":1}
I20260812 11:18:36.151126 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:36.150370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:46927)
0812 11:18:36.150854 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:36.150929 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":149,"threads_started":1}
I20260812 11:18:36.293076 20328 raft_consensus.cc:482] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:36.293349 20328 raft_consensus.cc:504] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:36.294055 20328 leader_election.cc:288] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839), e5131b92bf6545079aff20123940b83d (127.27.109.3:44629)
W20260812 11:18:36.295528 19632 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5131b92bf6545079aff20123940b83d (127.27.109.3:44629): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
W20260812 11:18:36.295784 19630 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:36.295866 19630 leader_election.cc:302] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1421b1e4e4c54c3e81ff323d791e0555; no voters: 4ec79a765d4d4a16b3ec7be780fec4e5, e5131b92bf6545079aff20123940b83d
I20260812 11:18:36.296173 20328 raft_consensus.cc:2726] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:18:36.449692 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:36.448934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:39725)
0812 11:18:36.449463 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:36.449561 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":139,"threads_started":1}
I20260812 11:18:36.881011 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:36.880154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:38355)
0812 11:18:36.880748 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:36.880862 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":211,"threads_started":1}
I20260812 11:18:37.153296 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:37.152528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:60015)
0812 11:18:37.152995 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:37.153102 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":106,"threads_started":1}
I20260812 11:18:37.451694 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:37.450874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:54731)
0812 11:18:37.451406 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:37.451531 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":123,"threads_started":1}
I20260812 11:18:37.883407 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:37.882660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:50291)
0812 11:18:37.883133 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:37.883260 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":98,"threads_started":1}
I20260812 11:18:38.155663 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:38.155006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:57385)
0812 11:18:38.155431 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:38.155538 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":114,"threads_started":1}
I20260812 11:18:38.198593 20342 raft_consensus.cc:482] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:38.198782 20342 raft_consensus.cc:504] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec79a765d4d4a16b3ec7be780fec4e5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 42839 } } peers { permanent_uuid: "e5131b92bf6545079aff20123940b83d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 44629 } } peers { permanent_uuid: "1421b1e4e4c54c3e81ff323d791e0555" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34219 } }
I20260812 11:18:38.199615 20342 leader_election.cc:288] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839), e5131b92bf6545079aff20123940b83d (127.27.109.3:44629)
W20260812 11:18:38.201174 19630 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ec79a765d4d4a16b3ec7be780fec4e5 (127.27.109.2:42839): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
W20260812 11:18:38.201512 19632 leader_election.cc:341] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5131b92bf6545079aff20123940b83d (127.27.109.3:44629): Not found: Tablet not found: 7e2f4f95117645ceaa80d7b463093347
I20260812 11:18:38.201663 19632 leader_election.cc:302] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1421b1e4e4c54c3e81ff323d791e0555; no voters: 4ec79a765d4d4a16b3ec7be780fec4e5, e5131b92bf6545079aff20123940b83d
I20260812 11:18:38.201989 20342 raft_consensus.cc:2726] T 7e2f4f95117645ceaa80d7b463093347 P 1421b1e4e4c54c3e81ff323d791e0555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:18:38.453591 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:38.452927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:34059)
0812 11:18:38.453361 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:38.453454 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":102,"threads_started":1}
I20260812 11:18:38.885345 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:38.884683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:55027)
0812 11:18:38.885114 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:38.885206 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
I20260812 11:18:39.157871 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:39.157113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:51289)
0812 11:18:39.157554 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:39.157685 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":94,"threads_started":1}
I20260812 11:18:39.455775 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:39.455063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:40259)
0812 11:18:39.455538 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:39.455642 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":123,"threads_started":1}
I20260812 11:18:39.887230 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:39.886511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:59239)
0812 11:18:39.886986 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:39.887086 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":103,"threads_started":1}
I20260812 11:18:40.160070 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:40.159364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:54593)
0812 11:18:40.159831 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:40.159932 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":119,"threads_started":1}
I20260812 11:18:40.457762 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:40.457094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:40553)
0812 11:18:40.457555 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:40.457642 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":111,"threads_started":1}
I20260812 11:18:40.889163 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:40.888543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:60757)
0812 11:18:40.888975 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:40.889052 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":117,"threads_started":1}
I20260812 11:18:41.162233 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:41.161448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:58507)
0812 11:18:41.161989 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:41.162093 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":115,"threads_started":1}
I20260812 11:18:41.460011 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:41.459142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:43865)
0812 11:18:41.459658 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:41.459752 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":191,"threads_started":1}
I20260812 11:18:41.891377 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:41.890670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.1:45325)
0812 11:18:41.891155 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:41.891229 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":201,"threads_started":1}
I20260812 11:18:42.164534 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:42.163874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.2:52345)
0812 11:18:42.164274 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:42.164381 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":97,"threads_started":1}
I20260812 11:18:42.425673 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 19551
I20260812 11:18:42.454015 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 19821
I20260812 11:18:42.462193 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:42.461591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:46227 (local address 127.27.109.3:60733)
0812 11:18:42.461991 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:42.462072 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260812 11:18:42.469785 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 20058
2026-08-12T11:18:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (21331 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:18:42 dist-test-slave-7z32 krb5kdc[20376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:18:42 dist-test-slave-7z32 krb5kdc[20376](info): set up 2 sockets
Aug 12 11:18:42 dist-test-slave-7z32 krb5kdc[20376](info): commencing operation
krb5kdc: starting...
W20260812 11:18:43.889991 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.377s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:18:44 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:18:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:18:44.081012 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39975
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:44.245085 20400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:44.245555 20400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:44.245618 20400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:44.251335 20400 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:18:44.251461 20400 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:18:44.251480 20400 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:18:44.251498 20400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:18:44.251523 20400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:44.251545 20400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:44.251610 20400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:18:44.251662 20400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:18:44.259069 20400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:44.260912 20400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:44.262511 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:44.274814 20406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:44.274961 20400 server_base.cc:1034] running on GCE node
W20260812 11:18:44.274900 20407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:44.274859 20409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:44.276032 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:44.277513 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:44.278748 20400 hybrid_clock.cc:648] HybridClock initialized: now 1786533524278726 us; error 49 us; skew 500 ppm
Aug 12 11:18:44 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:44.282361 20400 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:18:44.283646 20400 webserver.cc:460] Webserver started at http://127.27.109.62:44117/ using document root <none> and password file <none>
I20260812 11:18:44.284364 20400 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:44.284468 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:44.284744 20400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:44.287142 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "0ec05130ca604b85b843215609acbb6e"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "63858352619433a77ad2355b922456d5"
server_key_iv: "6266024cda315920acd807398b5ea282"
server_key_version: "encryptionkey@0"
I20260812 11:18:44.287794 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "0ec05130ca604b85b843215609acbb6e"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "63858352619433a77ad2355b922456d5"
server_key_iv: "6266024cda315920acd807398b5ea282"
server_key_version: "encryptionkey@0"
I20260812 11:18:44.292703 20400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:18:44.296057 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:44.298089 20400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:18:44.298309 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0ec05130ca604b85b843215609acbb6e"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "63858352619433a77ad2355b922456d5"
server_key_iv: "6266024cda315920acd807398b5ea282"
server_key_version: "encryptionkey@0"
I20260812 11:18:44.298529 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:44.324152 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:44.328922 20400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:44.329216 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:44.339988 20400 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39975
I20260812 11:18:44.340516 20498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39975 every 8 connection(s)
I20260812 11:18:44.341826 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:18:44.346160 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:44.346230 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 20400
I20260812 11:18:44.346477 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:18:44.347054 28084 external_mini_cluster.cc:1428] Setting key 49afa9784bbe198d50f81f71b80e7cff
I20260812 11:18:44.354943 20500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Bootstrap starting.
I20260812 11:18:44.358434 20500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:44.359760 20500 log.cc:824] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:18:44 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:44.363453 20500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: No bootstrap required, opened a new log
I20260812 11:18:44.368532 20500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:18:44.368854 20500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:44.368894 20500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec05130ca604b85b843215609acbb6e, State: Initialized, Role: FOLLOWER
I20260812 11:18:44.369451 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:44.349404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.27.109.62:39975)
0812 11:18:44.350040 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:44.350062 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:44.350112 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:18:44.351143 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:44.351173 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:44.351666 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:44.352088 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:44.353437 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:44.355020 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:44.356581 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:44.357318 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:44.357526 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:44.362345 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:44.362354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:44.362381 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:44.362427 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:44.364928 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:44.365726 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:44.365735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:44.365744 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:44.365917 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:44.366359 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:44.366362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:44.366363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:44.367052 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:44.367291 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:44.367656 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 11:18:44.368139 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":218,"threads_started":1}
I20260812 11:18:44.369459 20500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:18:44.369727 20500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:18:44.369787 20500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:18:44.369899 20500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:44.371167 20500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:18:44.371615 20500 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec05130ca604b85b843215609acbb6e; no voters:
I20260812 11:18:44.372063 20500 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:18:44.372159 20505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:44.372434 20505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 LEADER]: Becoming Leader. State: Replica: 0ec05130ca604b85b843215609acbb6e, State: Running, Role: LEADER
I20260812 11:18:44.372809 20505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:18:44.373704 20500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:18:44.375038 20511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec05130ca604b85b843215609acbb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } } }
I20260812 11:18:44.375299 20511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:44.375716 20506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec05130ca604b85b843215609acbb6e. Latest consensus state: current_term: 1 leader_uuid: "0ec05130ca604b85b843215609acbb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } } }
I20260812 11:18:44.375916 20524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:18:44.376178 20506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: This master's current role is: LEADER
I20260812 11:18:44.379796 20524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:18:44.386240 20524 catalog_manager.cc:1347] Generated new cluster ID: bc16c2323d454397b029e6a1babfd3c7
I20260812 11:18:44.386348 20524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:18:44.419756 20524 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:18:44.421613 20524 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:18:44.433322 20524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Generated new TSK 0
I20260812 11:18:44.434303 20524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:18:44.491938 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39975
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:44.683027 20546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:44.683481 20546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:44.683535 20546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:44.688371 20546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:44.688500 20546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:44.688578 20546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:18:44.695250 20546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:44.696765 20546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:44.698100 20546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:44.712883 20556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:44.712900 20559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:44.713207 20546 server_base.cc:1034] running on GCE node
W20260812 11:18:44.713069 20555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:44.714143 20546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:44.715116 20546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:44.716395 20546 hybrid_clock.cc:648] HybridClock initialized: now 1786533524716363 us; error 56 us; skew 500 ppm
Aug 12 11:18:44 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:44.720484 20546 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:18:44.721768 20546 webserver.cc:460] Webserver started at http://127.27.109.1:43509/ using document root <none> and password file <none>
I20260812 11:18:44.722512 20546 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:44.722594 20546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:44.722906 20546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:44.725414 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "1a61157fc09d491ca165ebd8ba700692"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "ecfe34bf6338f3b2e703a2be094b4051"
server_key_iv: "1cc3d66d93b8c159d33da0128597cbaf"
server_key_version: "encryptionkey@0"
I20260812 11:18:44.726114 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "1a61157fc09d491ca165ebd8ba700692"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "ecfe34bf6338f3b2e703a2be094b4051"
server_key_iv: "1cc3d66d93b8c159d33da0128597cbaf"
server_key_version: "encryptionkey@0"
I20260812 11:18:44.732659 20546 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260812 11:18:44.736618 20570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:44.738997 20546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260812 11:18:44.739248 20546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "1a61157fc09d491ca165ebd8ba700692"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "ecfe34bf6338f3b2e703a2be094b4051"
server_key_iv: "1cc3d66d93b8c159d33da0128597cbaf"
server_key_version: "encryptionkey@0"
I20260812 11:18:44.739444 20546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:44.756853 20546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:44.764422 20546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:44.764797 20546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:44.766024 20546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:44.767272 20546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:44.767340 20546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:44.767390 20546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:44.767405 20546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:44.781068 20546 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:39283
I20260812 11:18:44.781445 20759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:39283 every 8 connection(s)
I20260812 11:18:44.782346 20546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:18:44.787189 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 20546
I20260812 11:18:44.787421 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:18:44.787832 28084 external_mini_cluster.cc:1428] Setting key c6d41e954912d998cd29889423616a7b
Aug 12 11:18:44 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:18:44.804178 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:44.785148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:35149 (local address 127.27.109.62:39975)
0812 11:18:44.785374 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:44.785382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:44.786818 (+ 1436us) server_negotiation.cc:407] Connection header received
0812 11:18:44.788295 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:44.788298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:44.788371 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:44.788492 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:44.790004 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:44.790595 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:44.792027 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:44.792284 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:44.792412 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:44.797311 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:44.797320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:44.797327 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:44.797393 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:44.799979 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:44.801250 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:44.801256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:44.801258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:44.801401 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:44.802031 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:44.802035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:44.802036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:44.802350 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:44.802457 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:44.803554 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0812 11:18:44.803799 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:18:44.805048 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:44.785733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:35149)
0812 11:18:44.786559 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:44.786642 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:44.788010 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:44.788722 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:44.788731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:44.789193 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:44.789801 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:44.789817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:44.790764 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:44.790768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:44.791861 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:44.791875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:44.792489 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:44.792493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:44.792697 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:44.794150 (+ 1453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:44.794180 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:44.796923 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:44.800277 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:44.800291 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:44.800324 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:44.800964 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:44.801547 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:44.801551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:44.801554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:44.801840 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:44.802503 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:44.802512 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:44.803074 (+ 562us) client_negotiation.cc:769] Sending connection context
0812 11:18:44.803390 (+ 316us) client_negotiation.cc:240] Negotiation successful
0812 11:18:44.803881 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":223,"threads_started":1}
I20260812 11:18:44.807651 20761 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:18:44.808133 20761 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:44.809154 20761 heartbeater.cc:511] Master 127.27.109.62:39975 requested a full tablet report, sending...
I20260812 11:18:44.812261 20446 ts_manager.cc:194] Registered new tserver with Master: 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283)
I20260812 11:18:44.814635 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:35149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:18:44.835421 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39975
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:44.988628 20769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:44.988977 20769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:44.989027 20769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:44.994557 20769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:44.994685 20769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:44.994765 20769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:18:45.001770 20769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:45.003259 20769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:45.004505 20769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:45.032289 20781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:45.032245 20774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:45.032721 20769 server_base.cc:1034] running on GCE node
W20260812 11:18:45.032254 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:45.033331 20769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:45.034197 20769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:45.037698 20769 hybrid_clock.cc:648] HybridClock initialized: now 1786533525037686 us; error 112 us; skew 500 ppm
Aug 12 11:18:45 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533525, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:45.043066 20769 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:18:45.044447 20769 webserver.cc:460] Webserver started at http://127.27.109.2:33725/ using document root <none> and password file <none>
I20260812 11:18:45.045173 20769 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:45.045311 20769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:45.045660 20769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:45.048887 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "8e0909d21c5a4ddeb08a5af61289dd75"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "7db6d9c978bb8b5464eb9cdd13be45dc"
server_key_iv: "483fd857ad325c86608ba0b11030eba7"
server_key_version: "encryptionkey@0"
I20260812 11:18:45.049752 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "8e0909d21c5a4ddeb08a5af61289dd75"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "7db6d9c978bb8b5464eb9cdd13be45dc"
server_key_iv: "483fd857ad325c86608ba0b11030eba7"
server_key_version: "encryptionkey@0"
I20260812 11:18:45.056061 20769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 11:18:45.059868 20799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:45.062062 20769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:18:45.062318 20769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "8e0909d21c5a4ddeb08a5af61289dd75"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "7db6d9c978bb8b5464eb9cdd13be45dc"
server_key_iv: "483fd857ad325c86608ba0b11030eba7"
server_key_version: "encryptionkey@0"
I20260812 11:18:45.062511 20769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:45.139343 20769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:45.143565 20769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:45.143848 20769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:45.144644 20769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:45.145725 20769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:45.145821 20769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:45.145905 20769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:45.145952 20769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:45.158675 20769 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:37147
I20260812 11:18:45.158697 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:37147 every 8 connection(s)
I20260812 11:18:45.159785 20769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:18:45.160326 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 20769
I20260812 11:18:45.160501 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:18:45.161065 28084 external_mini_cluster.cc:1428] Setting key 579cf3e35291a17e4ec1b6f739946ff6
Aug 12 11:18:45 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533525, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260812 11:18:45.181313 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.162964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:45327 (local address 127.27.109.62:39975)
0812 11:18:45.163185 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.163191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.164742 (+ 1551us) server_negotiation.cc:407] Connection header received
0812 11:18:45.166077 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.166084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.166189 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.166365 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:45.168371 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.169021 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.170679 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.171227 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.171468 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.175487 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:45.175496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:45.175501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:45.175560 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:45.177654 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:45.178581 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:45.178587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:45.178589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:45.178693 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:45.179315 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:45.179317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:45.179319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:45.179675 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:45.179782 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.180841 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.181060 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:18:45.182341 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.163554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:45327)
0812 11:18:45.164496 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.164562 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:45.165747 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:45.166617 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:45.166625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:45.167203 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:45.167976 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.168004 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.169312 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.169316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.170330 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.170343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.171572 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.171576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.171785 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.172584 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:45.172605 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:45.175064 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:45.177918 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:45.177927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:45.177957 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:45.178399 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:45.178914 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:45.178918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:45.178919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:45.179132 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:45.179907 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:45.179914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:45.180426 (+ 512us) client_negotiation.cc:769] Sending connection context
0812 11:18:45.180709 (+ 283us) client_negotiation.cc:240] Negotiation successful
0812 11:18:45.181089 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":189,"threads_started":1}
I20260812 11:18:45.184294 20991 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39975
I20260812 11:18:45.184780 20991 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:45.185699 20991 heartbeater.cc:511] Master 127.27.109.62:39975 requested a full tablet report, sending...
I20260812 11:18:45.187676 20446 ts_manager.cc:194] Registered new tserver with Master: 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147)
I20260812 11:18:45.188511 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:45327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:18:45.208421 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39975
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:18:45.344694 20999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:18:45.345072 20999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:18:45.345153 20999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:18:45.350072 20999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:18:45.350186 20999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:18:45.350289 20999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:18:45.356092 20999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:18:45.357359 20999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:18:45.358486 20999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:18:45.369999 21007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:45.370010 21011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:18:45.370076 21006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:18:45.370170 20999 server_base.cc:1034] running on GCE node
I20260812 11:18:45.371472 20999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:18:45.372532 20999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:18:45.373835 20999 hybrid_clock.cc:648] HybridClock initialized: now 1786533525373820 us; error 70 us; skew 500 ppm
Aug 12 11:18:45 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533525, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:18:45.377882 20999 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:18:45.379230 20999 webserver.cc:460] Webserver started at http://127.27.109.3:35387/ using document root <none> and password file <none>
I20260812 11:18:45.380262 20999 fs_manager.cc:359] Metadata directory not provided
I20260812 11:18:45.380388 20999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:18:45.380666 20999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:18:45.383138 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "a48cf6c690514b0080ce2b4b6d6db621"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "2b77b8b0c5a03f3f59b52a1717ae802a"
server_key_iv: "e7d96a3dda5793186964b091dd1c6879"
server_key_version: "encryptionkey@0"
I20260812 11:18:45.383800 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "a48cf6c690514b0080ce2b4b6d6db621"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "2b77b8b0c5a03f3f59b52a1717ae802a"
server_key_iv: "e7d96a3dda5793186964b091dd1c6879"
server_key_version: "encryptionkey@0"
I20260812 11:18:45.388702 20999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:18:45.392731 21027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:45.395157 20999 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:18:45.395412 20999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "a48cf6c690514b0080ce2b4b6d6db621"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "2b77b8b0c5a03f3f59b52a1717ae802a"
server_key_iv: "e7d96a3dda5793186964b091dd1c6879"
server_key_version: "encryptionkey@0"
I20260812 11:18:45.395596 20999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:18:45.420567 20999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:18:45.424556 20999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:18:45.424804 20999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:18:45.425488 20999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:18:45.426469 20999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:18:45.426548 20999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:45.426622 20999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:18:45.426676 20999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:18:45.438397 20999 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38731
I20260812 11:18:45.438426 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38731 every 8 connection(s)
I20260812 11:18:45.439548 20999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:18:45.441731 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 20999
I20260812 11:18:45.441923 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:18:45.442314 28084 external_mini_cluster.cc:1428] Setting key 015d929aef8a1515739f003d3d84aa00
Aug 12 11:18:45 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533525, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:18:45.458405 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.441803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39107 (local address 127.27.109.62:39975)
0812 11:18:45.442031 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.442039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.443341 (+ 1302us) server_negotiation.cc:407] Connection header received
0812 11:18:45.444648 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.444652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.444726 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.444853 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:45.446499 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.447345 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.448700 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.448986 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.449111 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.452646 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:45.452650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:45.452653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:45.452691 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:45.454597 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:45.455711 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:45.455715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:45.455716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:45.455874 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:45.456500 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:45.456504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:45.456506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:45.456855 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:45.456996 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.458030 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.458249 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 11:18:45.459542 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.442213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:39107)
0812 11:18:45.443106 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.443165 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:45.444261 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:45.445074 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:45.445082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:45.445634 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:18:45.446211 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.446226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.447525 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.447528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.448518 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.448530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.449169 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.449172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.449344 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.450160 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:18:45.450178 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:18:45.452358 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:18:45.454897 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:45.454917 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:45.454976 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:45.455507 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:45.456034 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:18:45.456038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:18:45.456040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:18:45.456325 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:18:45.457097 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:18:45.457109 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:18:45.457638 (+ 529us) client_negotiation.cc:769] Sending connection context
0812 11:18:45.457923 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 11:18:45.458429 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":209,"threads_started":1}
I20260812 11:18:45.461532 21213 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39975
I20260812 11:18:45.461887 21213 heartbeater.cc:464] Registering TS with master...
I20260812 11:18:45.462538 21213 heartbeater.cc:511] Master 127.27.109.62:39975 requested a full tablet report, sending...
I20260812 11:18:45.464294 20446 ts_manager.cc:194] Registered new tserver with Master: a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731)
I20260812 11:18:45.465065 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:39107
I20260812 11:18:45.472846 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:18:45.486411 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.476304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.27.109.62:39975)
0812 11:18:45.476549 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.476555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.476795 (+ 240us) server_negotiation.cc:407] Connection header received
0812 11:18:45.476939 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.476942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.476999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.477113 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:18:45.478142 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.478733 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.480023 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.480269 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.480423 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.482322 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:18:45.482325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:18:45.482328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:18:45.482357 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:18:45.484058 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:45.484707 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:45.484710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:45.484711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:45.484809 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:18:45.485150 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:18:45.485152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:18:45.485153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:18:45.485470 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:18:45.485599 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.485998 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.486209 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 11:18:45.491123 20446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:18:45.494701 20446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:18:45.515331 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.508339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.27.109.3:38731)
0812 11:18:45.509425 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.509451 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.509490 (+ 39us) server_negotiation.cc:407] Connection header received
0812 11:18:45.509616 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.509623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.509945 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.510198 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:45.510599 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.512194 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.513830 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.514446 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.514581 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.514758 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.515040 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.515152 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":203,"threads_started":1}
I20260812 11:18:45.515321 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.508138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38731 (local address 127.0.0.1:57300)
0812 11:18:45.508906 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.508974 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:45.509122 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:45.510207 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:45.510213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:45.510245 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:45.510437 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.510446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.512372 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.512375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.513691 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.513702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.514600 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.514604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.514741 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.514792 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 11:18:45.514992 (+ 200us) client_negotiation.cc:240] Negotiation successful
0812 11:18:45.515152 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":25,"thread_start_us":240,"threads_started":1}
I20260812 11:18:45.516621 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.509417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37147 (local address 127.0.0.1:54744)
0812 11:18:45.509730 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.509749 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:45.509856 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:45.511360 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:45.511379 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:45.511403 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:45.511528 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.511535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.513157 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.513160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.514560 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.514568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.516070 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.516091 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.516213 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.516250 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:18:45.516313 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 11:18:45.516420 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":78,"thread_start_us":158,"threads_started":1}
I20260812 11:18:45.517781 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.509659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.27.109.2:37147)
0812 11:18:45.510765 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.510778 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.510804 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:18:45.510916 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.510922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.511164 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.511707 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:45.511877 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.513000 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.514784 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.515876 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.517172 (+ 1296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.517341 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.517422 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.517573 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":193,"threads_started":1}
I20260812 11:18:45.517941 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.509865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39283 (local address 127.0.0.1:57604)
0812 11:18:45.510240 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.510260 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:45.510370 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:45.511210 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:45.511213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:45.511237 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:45.511408 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.511422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.512961 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.512967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.514538 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.514553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.517406 (+ 2853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.517410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.517538 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.517565 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:18:45.517664 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:18:45.517763 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":42,"thread_start_us":116,"threads_started":1}
I20260812 11:18:45.518486 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.510190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.27.109.1:39283)
0812 11:18:45.510637 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.510643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.510667 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:18:45.510745 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.510749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.511034 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.511232 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:45.511553 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.512825 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.514755 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.516946 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.517102 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.517326 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.518151 (+ 825us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.518309 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":128,"threads_started":1}
I20260812 11:18:45.519243 21097 tablet_service.cc:1467] Processing CreateTablet for tablet aac34ef53dd64bf0b7da2f16850f76f8 (DEFAULT_TABLE table=test-table [id=fae4a690e3424130a5a77256754b4137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:45.520867 21097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac34ef53dd64bf0b7da2f16850f76f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:45.522089 20878 tablet_service.cc:1467] Processing CreateTablet for tablet aac34ef53dd64bf0b7da2f16850f76f8 (DEFAULT_TABLE table=test-table [id=fae4a690e3424130a5a77256754b4137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:45.522290 20649 tablet_service.cc:1467] Processing CreateTablet for tablet aac34ef53dd64bf0b7da2f16850f76f8 (DEFAULT_TABLE table=test-table [id=fae4a690e3424130a5a77256754b4137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:18:45.523533 20878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac34ef53dd64bf0b7da2f16850f76f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:45.523533 20649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac34ef53dd64bf0b7da2f16850f76f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:18:45.530340 21238 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Bootstrap starting.
I20260812 11:18:45.532500 21240 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap starting.
I20260812 11:18:45.535461 21242 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Bootstrap starting.
I20260812 11:18:45.536052 21240 tablet_bootstrap.cc:654] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:45.536862 21238 tablet_bootstrap.cc:654] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:45.537214 21240 log.cc:824] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:45.538697 21242 tablet_bootstrap.cc:654] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Neither blocks nor log segments found. Creating new log.
I20260812 11:18:45.539974 21238 log.cc:824] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:45.540251 21242 log.cc:824] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Log is configured to *not* fsync() on all Append() calls
I20260812 11:18:45.541677 21240 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: No bootstrap required, opened a new log
I20260812 11:18:45.542007 21240 ts_tablet_manager.cc:1397] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 11:18:45.543917 21238 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: No bootstrap required, opened a new log
I20260812 11:18:45.544248 21238 ts_tablet_manager.cc:1397] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260812 11:18:45.547013 21242 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: No bootstrap required, opened a new log
I20260812 11:18:45.547410 21242 ts_tablet_manager.cc:1397] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260812 11:18:45.547668 21240 raft_consensus.cc:348] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.548087 21240 raft_consensus.cc:374] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:45.548163 21240 raft_consensus.cc:729] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a61157fc09d491ca165ebd8ba700692, State: Initialized, Role: FOLLOWER
I20260812 11:18:45.548491 21238 raft_consensus.cc:348] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.548779 21238 raft_consensus.cc:374] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:45.548698 21240 consensus_queue.cc:260] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.548854 21238 raft_consensus.cc:729] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48cf6c690514b0080ce2b4b6d6db621, State: Initialized, Role: FOLLOWER
I20260812 11:18:45.549443 21238 consensus_queue.cc:260] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.549978 21240 ts_tablet_manager.cc:1428] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 11:18:45.550387 21238 ts_tablet_manager.cc:1428] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:18:45.550427 20761 heartbeater.cc:503] Master 127.27.109.62:39975 was elected leader, sending a full tablet report...
I20260812 11:18:45.551589 21213 heartbeater.cc:503] Master 127.27.109.62:39975 was elected leader, sending a full tablet report...
I20260812 11:18:45.553148 21242 raft_consensus.cc:348] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.553661 21242 raft_consensus.cc:374] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:18:45.553740 21242 raft_consensus.cc:729] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0909d21c5a4ddeb08a5af61289dd75, State: Initialized, Role: FOLLOWER
I20260812 11:18:45.554358 21242 consensus_queue.cc:260] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.555405 20991 heartbeater.cc:503] Master 127.27.109.62:39975 was elected leader, sending a full tablet report...
I20260812 11:18:45.555845 21242 ts_tablet_manager.cc:1428] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 11:18:45.640954 21249 raft_consensus.cc:482] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:18:45.641342 21249 raft_consensus.cc:504] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.643018 21249 leader_election.cc:288] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283)
I20260812 11:18:45.649822 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.643544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37147 (local address 127.27.109.3:34515)
0812 11:18:45.643744 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.643771 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:45.643956 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:45.644602 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:45.644606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:45.644638 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:45.644831 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.644843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.645972 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.645975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.647572 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.647590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.648969 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.648973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.649220 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.649270 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 11:18:45.649416 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 11:18:45.649582 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260812 11:18:45.649935 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.643676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:34515 (local address 127.27.109.2:37147)
0812 11:18:45.644082 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.644089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.644120 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:18:45.644218 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.644222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.644301 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.644442 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:45.645059 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.645787 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.647828 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.648614 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.648828 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.648898 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.649635 (+ 737us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.649753 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260812 11:18:45.649953 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.644096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:49589 (local address 127.27.109.1:39283)
0812 11:18:45.644299 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.644304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.645037 (+ 733us) server_negotiation.cc:407] Connection header received
0812 11:18:45.645117 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.645120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.645183 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.645296 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:18:45.645895 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.646475 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.648357 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.649039 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.649159 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.649201 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.649635 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.649753 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:18:45.649832 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.644051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39283 (local address 127.27.109.3:49589)
0812 11:18:45.644761 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.644784 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:18:45.644928 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:18:45.645462 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:18:45.645467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:18:45.645556 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:18:45.645719 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.645727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.646785 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.646790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.648020 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:18:45.648029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.649222 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.649226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:18:45.649348 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.649375 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:18:45.649467 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:18:45.649583 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":71,"thread_start_us":337,"threads_started":1}
I20260812 11:18:45.651499 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac34ef53dd64bf0b7da2f16850f76f8" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a61157fc09d491ca165ebd8ba700692" is_pre_election: true
I20260812 11:18:45.651439 20910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac34ef53dd64bf0b7da2f16850f76f8" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" is_pre_election: true
I20260812 11:18:45.651885 20680 raft_consensus.cc:2454] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 0.
I20260812 11:18:45.651885 20910 raft_consensus.cc:2454] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 0.
I20260812 11:18:45.652626 21030 leader_election.cc:302] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a61157fc09d491ca165ebd8ba700692, a48cf6c690514b0080ce2b4b6d6db621; no voters:
I20260812 11:18:45.652973 21249 raft_consensus.cc:2781] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:18:45.653081 21249 raft_consensus.cc:482] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:18:45.653131 21249 raft_consensus.cc:3037] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:45.654318 21249 raft_consensus.cc:504] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:18:45.654757 21249 leader_election.cc:288] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283)
I20260812 11:18:45.655547 20910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac34ef53dd64bf0b7da2f16850f76f8" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0909d21c5a4ddeb08a5af61289dd75"
I20260812 11:18:45.655547 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac34ef53dd64bf0b7da2f16850f76f8" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a61157fc09d491ca165ebd8ba700692"
I20260812 11:18:45.655822 20680 raft_consensus.cc:3037] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:45.655822 20910 raft_consensus.cc:3037] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:18:45.657732 20910 raft_consensus.cc:2454] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 1.
I20260812 11:18:45.657732 20680 raft_consensus.cc:2454] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 1.
I20260812 11:18:45.658401 21030 leader_election.cc:302] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a61157fc09d491ca165ebd8ba700692, a48cf6c690514b0080ce2b4b6d6db621; no voters:
I20260812 11:18:45.658778 21249 raft_consensus.cc:2781] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:18:45.659188 21249 raft_consensus.cc:686] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 LEADER]: Becoming Leader. State: Replica: a48cf6c690514b0080ce2b4b6d6db621, State: Running, Role: LEADER
I20260812 11:18:45.659759 21249 consensus_queue.cc:237] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
W20260812 11:18:45.661692 20992 tablet.cc:2369] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:45.665319 20446 catalog_manager.cc:5495] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 reported cstate change: term changed from 0 to 1, leader changed from <none> to a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "a48cf6c690514b0080ce2b4b6d6db621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } health_report { overall_health: UNKNOWN } } }
W20260812 11:18:45.691258 21215 tablet.cc:2369] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:18:45.720722 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.715641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.27.109.3:38731)
0812 11:18:45.715795 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 11:18:45.715798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:18:45.715816 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:18:45.715925 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:18:45.715929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:18:45.716005 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:18:45.716153 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:18:45.716622 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.717187 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.718520 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:18:45.718780 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:18:45.719003 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:18:45.719528 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:18:45.720078 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:18:45.720206 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:18:45.720487 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 11:18:45.720572 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:18:45.729988 20680 raft_consensus.cc:1261] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Refusing update from remote peer a48cf6c690514b0080ce2b4b6d6db621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:45.729988 20910 raft_consensus.cc:1261] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Refusing update from remote peer a48cf6c690514b0080ce2b4b6d6db621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:18:45.731217 21249 consensus_queue.cc:1035] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:45.731642 21255 consensus_queue.cc:1035] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:18:45.741412 21259 mvcc.cc:205] Tried to move back new op lower bound from 7317641321377734656 to 7317641321105985536. Current Snapshot: MvccSnapshot[applied={T|T < 7317641321377734656}]
I20260812 11:18:45.745970 21260 mvcc.cc:205] Tried to move back new op lower bound from 7317641321377734656 to 7317641321105985536. Current Snapshot: MvccSnapshot[applied={T|T < 7317641321377734656 or (T in {7317641321377734656})}]
I20260812 11:18:45.753623 20445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:45.753935 20445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:18:45.757738 20445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Sending DeleteTablet for 3 replicas of tablet aac34ef53dd64bf0b7da2f16850f76f8
I20260812 11:18:45.759023 20878 tablet_service.cc:1514] Processing DeleteTablet for tablet aac34ef53dd64bf0b7da2f16850f76f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:45 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:54744
I20260812 11:18:45.759023 21097 tablet_service.cc:1514] Processing DeleteTablet for tablet aac34ef53dd64bf0b7da2f16850f76f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:45 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57300
I20260812 11:18:45.759114 20649 tablet_service.cc:1514] Processing DeleteTablet for tablet aac34ef53dd64bf0b7da2f16850f76f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:45 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57604
I20260812 11:18:45.759876 21270 tablet_replica.cc:330] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: stopping tablet replica
I20260812 11:18:45.759865 21273 tablet_replica.cc:330] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: stopping tablet replica
I20260812 11:18:45.760231 21272 tablet_replica.cc:330] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: stopping tablet replica
I20260812 11:18:45.760329 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 20546
I20260812 11:18:45.777169 21270 raft_consensus.cc:2229] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:18:45.777976 21270 raft_consensus.cc:2258] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:45.778738 21272 raft_consensus.cc:2229] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:18:45.779338 21272 raft_consensus.cc:2258] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:18:45.783113 21272 ts_tablet_manager.cc:1905] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:45.792182 21270 ts_tablet_manager.cc:1905] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:18:45.797039 21270 ts_tablet_manager.cc:1918] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:45.797250 21270 log.cc:1196] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/aac34ef53dd64bf0b7da2f16850f76f8
I20260812 11:18:45.797765 21270 ts_tablet_manager.cc:1939] T aac34ef53dd64bf0b7da2f16850f76f8 P a48cf6c690514b0080ce2b4b6d6db621: Deleting consensus metadata
I20260812 11:18:45.799832 20424 catalog_manager.cc:4841] TS a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731): tablet aac34ef53dd64bf0b7da2f16850f76f8 (table test-table [id=fae4a690e3424130a5a77256754b4137]) successfully deleted
I20260812 11:18:45.800423 21272 ts_tablet_manager.cc:1918] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:18:45.800603 21272 log.cc:1196] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/aac34ef53dd64bf0b7da2f16850f76f8
I20260812 11:18:45.801147 21272 ts_tablet_manager.cc:1939] T aac34ef53dd64bf0b7da2f16850f76f8 P 8e0909d21c5a4ddeb08a5af61289dd75: Deleting consensus metadata
I20260812 11:18:45.803094 20424 catalog_manager.cc:4841] TS 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147): tablet aac34ef53dd64bf0b7da2f16850f76f8 (table test-table [id=fae4a690e3424130a5a77256754b4137]) successfully deleted
W20260812 11:18:45.827646 20424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:39283 (error 108)
I20260812 11:18:45.827913 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 20769
I20260812 11:18:45.889338 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:18:45.848634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39283 (local address 127.0.0.1:57620)
0812 11:18:45.888914 (+ 40280us) negotiation.cc:107] Waiting for socket to connect
0812 11:18:45.889160 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:39283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40074}
W20260812 11:18:45.889837 20424 catalog_manager.cc:4584] TS 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aac34ef53dd64bf0b7da2f16850f76f8: Network error: Client connection negotiation failed: client connection to 127.27.109.1:39283: connect: Connection refused (error 111)
I20260812 11:18:45.899514 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 20999
I20260812 11:18:45.922889 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 20400
I20260812 11:19:05.944204 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39975
--webserver_interface=127.27.109.62
--webserver_port=44117
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:06.081616 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:06.081983 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:06.082032 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:06.086947 21276 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:19:06.087028 21276 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:19:06.087046 21276 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:19:06.087064 21276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:06.087086 21276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:06.087108 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:06.087127 21276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:06.087145 21276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:06.093565 21276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:06.095113 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:06.096709 21276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:06.115867 21285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:06.115926 21286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:06.116114 21276 server_base.cc:1034] running on GCE node
W20260812 11:19:06.116329 21291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:06.117305 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:06.119061 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:06.120353 21276 hybrid_clock.cc:648] HybridClock initialized: now 1786533546120331 us; error 49 us; skew 500 ppm
Aug 12 11:19:06 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:06.124884 21276 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:19:06.126564 21276 webserver.cc:460] Webserver started at http://127.27.109.62:44117/ using document root <none> and password file <none>
I20260812 11:19:06.127506 21276 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:06.127664 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:06.134105 21276 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:19:06.137727 21300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.139971 21276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:19:06.140189 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0ec05130ca604b85b843215609acbb6e"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "63858352619433a77ad2355b922456d5"
server_key_iv: "6266024cda315920acd807398b5ea282"
server_key_version: "encryptionkey@0"
I20260812 11:19:06.141028 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:06.163695 21276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:06.167737 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:06.168025 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:06.177500 21276 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39975
I20260812 11:19:06.177832 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39975 every 8 connection(s)
I20260812 11:19:06.179035 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:06.186589 21388 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:19:06.187875 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 21276
I20260812 11:19:06.188580 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:39283
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:39975
--webserver_port=43509
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:19:06.190205 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Bootstrap starting.
I20260812 11:19:06.217628 21388 log.cc:824] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:06.231747 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:19:06.232376 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Bootstrap complete.
I20260812 11:19:06.238301 21388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:06.239009 21388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec05130ca604b85b843215609acbb6e, State: Initialized, Role: FOLLOWER
I20260812 11:19:06.240068 21388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:06.240351 21388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:06.240492 21388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:06.240650 21388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:19:06.243734 21388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:06.244259 21388 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec05130ca604b85b843215609acbb6e; no voters:
I20260812 11:19:06.244874 21388 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:19:06.245004 21394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:19:06.245275 21394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 LEADER]: Becoming Leader. State: Replica: 0ec05130ca604b85b843215609acbb6e, State: Running, Role: LEADER
I20260812 11:19:06.245837 21394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:06.246685 21388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:06.247970 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec05130ca604b85b843215609acbb6e. Latest consensus state: current_term: 2 leader_uuid: "0ec05130ca604b85b843215609acbb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } } }
I20260812 11:19:06.248167 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:06.248035 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0ec05130ca604b85b843215609acbb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } } }
I20260812 11:19:06.248248 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:06.250202 21404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:19:06.256215 21404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:19:06.257267 21404 catalog_manager.cc:1259] Loaded cluster ID: bc16c2323d454397b029e6a1babfd3c7
I20260812 11:19:06.257397 21404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:19:06.260543 21404 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:19:06.261876 21404 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Loaded TSK: 0
I20260812 11:19:06.279722 21404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Generated new TSK 1
I20260812 11:19:06.280195 21404 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:19:06.367985 21391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:06.368464 21391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:06.368557 21391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:06.373754 21391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:06.373883 21391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:06.373991 21391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:06.380882 21391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:39283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=43509
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:06.382695 21391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:06.384299 21391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:06.399147 21430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:06.399154 21426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:06.399220 21427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:06.400269 21391 server_base.cc:1034] running on GCE node
I20260812 11:19:06.400947 21391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:06.402081 21391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:06.405525 21391 hybrid_clock.cc:648] HybridClock initialized: now 1786533546405430 us; error 108 us; skew 500 ppm
Aug 12 11:19:06 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:06.411425 21391 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:19:06.413398 21391 webserver.cc:460] Webserver started at http://127.27.109.1:43509/ using document root <none> and password file <none>
I20260812 11:19:06.414601 21391 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:06.414889 21391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:06.421452 21391 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 11:19:06.425778 21442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.428225 21391 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 11:19:06.428525 21391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "1a61157fc09d491ca165ebd8ba700692"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "ecfe34bf6338f3b2e703a2be094b4051"
server_key_iv: "1cc3d66d93b8c159d33da0128597cbaf"
server_key_version: "encryptionkey@0"
I20260812 11:19:06.429468 21391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:06.450346 21391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:06.455093 21391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:06.455376 21391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:06.456347 21391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:06.458024 21451 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:19:06.463253 21391 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:19:06.463461 21391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260812 11:19:06.463660 21391 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:19:06.466383 21391 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:19:06.466482 21391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260812 11:19:06.467017 21451 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap starting.
I20260812 11:19:06.480647 21391 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:39283
I20260812 11:19:06.480742 21629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:39283 every 8 connection(s)
I20260812 11:19:06.482077 21391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:06.485458 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 21391
I20260812 11:19:06.486248 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:37147
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:39975
--webserver_port=33725
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:19:06.490687 21451 log.cc:824] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:06.499750 21451 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:19:06.500249 21451 tablet_bootstrap.cc:492] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap complete.
Aug 12 11:19:06 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:06.501819 21451 ts_tablet_manager.cc:1397] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Time spent bootstrapping tablet: real 0.035s user 0.032s sys 0.000s
I20260812 11:19:06.504984 21451 raft_consensus.cc:348] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:06.505535 21451 raft_consensus.cc:729] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a61157fc09d491ca165ebd8ba700692, State: Initialized, Role: FOLLOWER
I20260812 11:19:06.506145 21451 consensus_queue.cc:260] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:06.507295 21451 ts_tablet_manager.cc:1428] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260812 11:19:06.509792 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:06.484981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:33465)
0812 11:19:06.486148 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:06.486215 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:06.487604 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:06.489453 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:06.489466 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:06.490088 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:06.490933 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:06.490960 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.493997 (+ 3037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.494002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:06.495484 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:06.495501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.497324 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.497329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:06.497575 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:06.498572 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:06.498594 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:06.501088 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:06.505478 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:06.505489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:06.505508 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:06.506118 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:06.506729 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:06.506734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:06.506737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:06.507025 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:06.507992 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:06.507999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:06.508311 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 11:19:06.508602 (+ 291us) client_negotiation.cc:240] Negotiation successful
0812 11:19:06.508939 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":207,"threads_started":1}
I20260812 11:19:06.510907 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:06.485174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33465 (local address 127.27.109.62:39975)
0812 11:19:06.485783 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:06.485807 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:06.486368 (+ 561us) server_negotiation.cc:407] Connection header received
0812 11:19:06.488246 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:06.488260 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:06.488740 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:06.489259 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:06.491286 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.493680 (+ 2394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.495761 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.497025 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.497370 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:06.501598 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:06.501614 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:06.501653 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:06.501734 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:06.505251 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:06.506324 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:06.506333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:06.506344 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:06.506576 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:06.507173 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:06.507176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:06.507178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:06.507804 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:06.507969 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:06.508818 (+ 849us) server_negotiation.cc:299] Negotiation successful
0812 11:19:06.509279 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":164,"threads_started":1}
I20260812 11:19:06.514261 21630 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39975
I20260812 11:19:06.514652 21630 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:06.515508 21630 heartbeater.cc:511] Master 127.27.109.62:39975 requested a full tablet report, sending...
I20260812 11:19:06.518770 21326 ts_manager.cc:194] Registered new tserver with Master: 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283)
I20260812 11:19:06.523280 21326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:33465
I20260812 11:19:06.526613 21630 heartbeater.cc:503] Master 127.27.109.62:39975 was elected leader, sending a full tablet report...
Aug 12 11:19:06 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260812 11:19:06.538452 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:06.522499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39283 (local address 127.0.0.1:45512)
0812 11:19:06.523835 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:06.523918 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:06.524188 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:06.525152 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:06.525158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:06.525725 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:06.525910 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:06.525921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.527374 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.527382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:06.528352 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:06.528366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.529383 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.529386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:06.529496 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:06.530241 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:06.530255 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:06.532632 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:06.535924 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:06.535979 (+ 55us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:06.535988 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:06.536395 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:06.536839 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:06.536843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:06.536845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:06.536940 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:06.537548 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:06.537556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:06.537705 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 11:19:06.537880 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 11:19:06.538198 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":157,"threads_started":1}
I20260812 11:19:06.538437 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:06.522772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.27.109.1:39283)
0812 11:19:06.523558 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:06.523573 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:06.524111 (+ 538us) server_negotiation.cc:407] Connection header received
0812 11:19:06.524469 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:06.524477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:06.524733 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:06.524886 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:06.526104 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.527194 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.528538 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.529151 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.529290 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:06.532935 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:06.532962 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:06.532985 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:06.533040 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:06.535676 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:06.536585 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:06.536593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:06.536594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:06.536717 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:06.537051 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:06.537054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:06.537056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:06.537408 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:06.537585 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:06.537980 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 11:19:06.538218 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":193,"threads_started":1}
I20260812 11:19:06.540201 21518 tablet_service.cc:1514] Processing DeleteTablet for tablet aac34ef53dd64bf0b7da2f16850f76f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:18:45 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45512
I20260812 11:19:06.541936 21643 tablet_replica.cc:330] stopping tablet replica
I20260812 11:19:06.542439 21643 raft_consensus.cc:2229] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:06.542807 21643 raft_consensus.cc:2258] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:06.544219 21643 ts_tablet_manager.cc:1905] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:06.549816 21643 ts_tablet_manager.cc:1918] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:06.549984 21643 log.cc:1196] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal/wals/aac34ef53dd64bf0b7da2f16850f76f8
I20260812 11:19:06.550324 21643 ts_tablet_manager.cc:1939] T aac34ef53dd64bf0b7da2f16850f76f8 P 1a61157fc09d491ca165ebd8ba700692: Deleting consensus metadata
I20260812 11:19:06.552757 21306 catalog_manager.cc:4841] TS 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283): tablet aac34ef53dd64bf0b7da2f16850f76f8 (table test-table [id=fae4a690e3424130a5a77256754b4137]) successfully deleted
W20260812 11:19:06.658321 21636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:06.658742 21636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:06.658797 21636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:06.663790 21636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:06.663921 21636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:06.664000 21636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:06.671022 21636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=33725
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:06.672772 21636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:06.674978 21636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:06.688884 21650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:06.688937 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:06.689515 21636 server_base.cc:1034] running on GCE node
W20260812 11:19:06.688887 21651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:06.690163 21636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:06.691125 21636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:06.692693 21636 hybrid_clock.cc:648] HybridClock initialized: now 1786533546692727 us; error 133 us; skew 500 ppm
Aug 12 11:19:06 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:06.703326 21636 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:19:06.704871 21636 webserver.cc:460] Webserver started at http://127.27.109.2:33725/ using document root <none> and password file <none>
I20260812 11:19:06.705639 21636 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:06.705778 21636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:06.711346 21636 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:19:06.714565 21662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.716776 21636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:19:06.717000 21636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "8e0909d21c5a4ddeb08a5af61289dd75"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "7db6d9c978bb8b5464eb9cdd13be45dc"
server_key_iv: "483fd857ad325c86608ba0b11030eba7"
server_key_version: "encryptionkey@0"
I20260812 11:19:06.717720 21636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:06.736650 21636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:06.740499 21636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:06.740737 21636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:06.741438 21636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:06.742484 21636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:06.742568 21636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.742645 21636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:06.742692 21636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.755512 21636 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:37147
I20260812 11:19:06.755535 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:37147 every 8 connection(s)
I20260812 11:19:06.756764 21636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:06.760365 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 21636
I20260812 11:19:06.761026 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:38731
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:39975
--webserver_port=35387
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:19:06 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:06.776500 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:06.759664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34471 (local address 127.27.109.62:39975)
0812 11:19:06.759859 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:06.759864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:06.761718 (+ 1854us) server_negotiation.cc:407] Connection header received
0812 11:19:06.763093 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:06.763098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:06.763187 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:06.763315 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:06.765085 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.765713 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.767263 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.767555 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.767697 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:06.771244 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:06.771248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:06.771250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:06.771289 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:06.773145 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:06.774054 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:06.774058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:06.774059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:06.774183 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:06.774696 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:06.774705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:06.774707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:06.775113 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:06.775260 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:06.776016 (+ 756us) server_negotiation.cc:299] Negotiation successful
0812 11:19:06.776255 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:19:06.777953 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:06.760227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:34471)
0812 11:19:06.761461 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:06.761533 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:06.762719 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:06.763591 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:06.763601 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:06.764151 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:06.764847 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:06.764866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.765896 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.765900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:06.767089 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:06.767104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:06.767769 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:06.767774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:06.768001 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:06.768852 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:06.768875 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:06.770942 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:06.773347 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:06.773358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:06.773392 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:06.773898 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:06.774314 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:06.774319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:06.774321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:06.774560 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:06.775260 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:06.775272 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:06.775687 (+ 415us) client_negotiation.cc:769] Sending connection context
0812 11:19:06.775952 (+ 265us) client_negotiation.cc:240] Negotiation successful
0812 11:19:06.776447 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":218,"threads_started":1}
I20260812 11:19:06.780001 21837 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39975
I20260812 11:19:06.780408 21837 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:06.781561 21837 heartbeater.cc:511] Master 127.27.109.62:39975 requested a full tablet report, sending...
I20260812 11:19:06.783795 21326 ts_manager.cc:194] Registered new tserver with Master: 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147)
I20260812 11:19:06.784754 21326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:34471
W20260812 11:19:06.907831 21841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:06.908179 21841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:06.908231 21841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:06.913321 21841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:06.913432 21841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:06.913550 21841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:06.919368 21841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:38731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=35387
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:06.920689 21841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:06.921955 21841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:06.934317 21854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:06.934573 21841 server_base.cc:1034] running on GCE node
W20260812 11:19:06.934306 21848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:06.934306 21849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:06.935557 21841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:06.936693 21841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:06.937937 21841 hybrid_clock.cc:648] HybridClock initialized: now 1786533546937927 us; error 65 us; skew 500 ppm
Aug 12 11:19:06 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:06.942041 21841 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:19:06.943524 21841 webserver.cc:460] Webserver started at http://127.27.109.3:35387/ using document root <none> and password file <none>
I20260812 11:19:06.944312 21841 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:06.944417 21841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:06.950610 21841 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:19:06.954035 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.956501 21841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:19:06.956777 21841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "a48cf6c690514b0080ce2b4b6d6db621"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "2b77b8b0c5a03f3f59b52a1717ae802a"
server_key_iv: "e7d96a3dda5793186964b091dd1c6879"
server_key_version: "encryptionkey@0"
I20260812 11:19:06.957690 21841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:06.973097 21841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:06.978991 21841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:06.979246 21841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:06.980129 21841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:06.981447 21841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:06.981518 21841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.981582 21841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:06.981612 21841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:06.997210 21841 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38731
I20260812 11:19:06.997231 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38731 every 8 connection(s)
I20260812 11:19:06.999102 21841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:07.002691 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 21841
Aug 12 11:19:07 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:07.016327 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.003894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.27.109.62:39975)
0812 11:19:07.004839 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.004846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.004863 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:19:07.004913 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.004917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.004973 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.005090 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:07.006468 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.007366 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.008958 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.009319 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.009482 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.011670 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:07.011674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:07.011677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:07.011715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:07.014030 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:07.014612 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:07.014616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:07.014618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:07.014764 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:07.015156 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:07.015159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:07.015161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:07.015426 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:07.015750 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.015872 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.016098 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"spinlock_wait_cycles":1280,"thread_start_us":177,"threads_started":1}
I20260812 11:19:07.019709 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.002280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:49979 (local address 127.27.109.62:39975)
0812 11:19:07.002563 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.002572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.003937 (+ 1365us) server_negotiation.cc:407] Connection header received
0812 11:19:07.005477 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.005483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.005574 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.005748 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:07.007718 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.008620 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.010528 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.010767 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.010961 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.014666 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:07.014670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:07.014673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:07.014720 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:07.016717 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:07.017467 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:07.017471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:07.017473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:07.017569 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:07.017984 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:07.017987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:07.017989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:07.018242 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:07.018413 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.019327 (+ 914us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.019512 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 11:19:07.020691 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.002659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:49979)
0812 11:19:07.003710 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:07.003783 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:07.005163 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:07.006030 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:07.006039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:07.006678 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:07.007486 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.007513 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.008834 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.008837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.010361 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.010379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.011025 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.011030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.011232 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.011898 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:07.011917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:07.014404 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:07.016873 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:07.016882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:07.016905 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:07.017343 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:07.017673 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:07.017676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:07.017678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:07.017859 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:07.018413 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:07.018423 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:07.018998 (+ 575us) client_negotiation.cc:769] Sending connection context
0812 11:19:07.019274 (+ 276us) client_negotiation.cc:240] Negotiation successful
0812 11:19:07.019704 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":263,"threads_started":1}
I20260812 11:19:07.022485 22055 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39975
I20260812 11:19:07.023074 22055 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:07.023943 22055 heartbeater.cc:511] Master 127.27.109.62:39975 requested a full tablet report, sending...
I20260812 11:19:07.025828 21326 ts_manager.cc:194] Registered new tserver with Master: a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731)
I20260812 11:19:07.026587 21326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:49979
I20260812 11:19:07.032301 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:19:07.038172 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.033440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.27.109.62:39975)
0812 11:19:07.033621 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.033625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.033902 (+ 277us) server_negotiation.cc:407] Connection header received
0812 11:19:07.034075 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.034078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.034207 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.034330 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:07.034768 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.035500 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.036795 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.037031 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.037140 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.037586 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:07.037770 (+ 184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 11:19:07.038044 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43726: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 11:19:07.038426 21635 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43726: token expired
W20260812 11:19:07.039125 21220 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.109.62:39975)
I20260812 11:19:07.051199 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.039722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.27.109.62:39975)
0812 11:19:07.039883 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.039887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.039911 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:19:07.039979 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.039981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.040039 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.040132 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:07.041168 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.041768 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.043383 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.043636 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.043759 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.046217 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:07.046223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:07.046226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:07.046276 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:07.048646 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:07.049455 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:07.049458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:07.049459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:07.049563 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:07.049986 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:07.049989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:07.049991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:07.050243 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:07.050387 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.050891 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.051058 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:19:07.055011 21326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:19:07.059042 21326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:19:07.072340 21518 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7d9e3441ad4e6a9f4644673979b4a4 (DEFAULT_TABLE table=test-table [id=509e307453c7459398a449d72db37b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:07.073395 21518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7d9e3441ad4e6a9f4644673979b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:07.078186 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.070925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38731 (local address 127.0.0.1:53280)
0812 11:19:07.071229 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:07.071247 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:07.071349 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:07.072773 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:07.072776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:07.072794 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:07.072903 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.072910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.074484 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.074488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.076024 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.076038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.077555 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.077559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.077686 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.077755 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 11:19:07.077863 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 11:19:07.077983 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260812 11:19:07.078447 22080 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap starting.
I20260812 11:19:07.078577 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.070326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37147 (local address 127.0.0.1:36022)
0812 11:19:07.070696 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:07.070713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:07.070897 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:07.072512 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:07.072518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:07.072544 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:07.072789 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.072797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.074643 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.074647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.076183 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.076193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.077911 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.077916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.078062 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.078106 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 11:19:07.078220 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 11:19:07.078359 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":179,"threads_started":1}
I20260812 11:19:07.079102 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.070654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.27.109.2:37147)
0812 11:19:07.071667 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.071689 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.071725 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:19:07.071854 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.071863 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.072211 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.072451 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:07.073028 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.074458 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.076376 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.077584 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.077709 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.077917 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.078697 (+ 780us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.078933 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":224,"threads_started":1}
I20260812 11:19:07.079607 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.071152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.27.109.3:38731)
0812 11:19:07.071927 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.071945 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.071973 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:19:07.072089 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.072096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.072573 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.073055 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:07.073212 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.074323 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.076251 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.077354 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.077523 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.077759 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.078991 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.079393 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":299,"threads_started":1}
I20260812 11:19:07.080212 22080 tablet_bootstrap.cc:654] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:07.083125 21951 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7d9e3441ad4e6a9f4644673979b4a4 (DEFAULT_TABLE table=test-table [id=509e307453c7459398a449d72db37b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:07.083093 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7d9e3441ad4e6a9f4644673979b4a4 (DEFAULT_TABLE table=test-table [id=509e307453c7459398a449d72db37b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:07.084558 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7d9e3441ad4e6a9f4644673979b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:07.084599 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7d9e3441ad4e6a9f4644673979b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:07.086035 22080 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: No bootstrap required, opened a new log
I20260812 11:19:07.086177 22080 ts_tablet_manager.cc:1397] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260812 11:19:07.086766 22080 raft_consensus.cc:348] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.086969 22080 raft_consensus.cc:374] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:07.087020 22080 raft_consensus.cc:729] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a61157fc09d491ca165ebd8ba700692, State: Initialized, Role: FOLLOWER
I20260812 11:19:07.087193 22080 consensus_queue.cc:260] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.087651 22080 ts_tablet_manager.cc:1428] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 11:19:07.095012 22086 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Bootstrap starting.
I20260812 11:19:07.095309 22087 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Bootstrap starting.
I20260812 11:19:07.098464 22086 tablet_bootstrap.cc:654] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:07.099349 22087 tablet_bootstrap.cc:654] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:07.099898 22086 log.cc:824] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:07.100361 22087 log.cc:824] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:07.102298 22086 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: No bootstrap required, opened a new log
I20260812 11:19:07.102591 22086 ts_tablet_manager.cc:1397] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:19:07.103549 22087 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: No bootstrap required, opened a new log
I20260812 11:19:07.103927 22087 ts_tablet_manager.cc:1397] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 11:19:07.106603 22086 raft_consensus.cc:348] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.107033 22086 raft_consensus.cc:374] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:07.107117 22086 raft_consensus.cc:729] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48cf6c690514b0080ce2b4b6d6db621, State: Initialized, Role: FOLLOWER
I20260812 11:19:07.107612 22086 consensus_queue.cc:260] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.108582 22055 heartbeater.cc:503] Master 127.27.109.62:39975 was elected leader, sending a full tablet report...
I20260812 11:19:07.108841 22086 ts_tablet_manager.cc:1428] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260812 11:19:07.108997 22087 raft_consensus.cc:348] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.109460 22087 raft_consensus.cc:374] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:07.109534 22087 raft_consensus.cc:729] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0909d21c5a4ddeb08a5af61289dd75, State: Initialized, Role: FOLLOWER
I20260812 11:19:07.110212 22087 consensus_queue.cc:260] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.111418 21837 heartbeater.cc:503] Master 127.27.109.62:39975 was elected leader, sending a full tablet report...
I20260812 11:19:07.111804 22087 ts_tablet_manager.cc:1428] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260812 11:19:07.234032 21631 tablet.cc:2369] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:07.250852 22056 tablet.cc:2369] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:07.258383 21838 tablet.cc:2369] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:07.273427 22092 raft_consensus.cc:482] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:07.273949 22092 raft_consensus.cc:504] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.275792 22092 leader_election.cc:288] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283)
I20260812 11:19:07.281356 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.276302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37475 (local address 127.27.109.2:37147)
0812 11:19:07.276615 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.276622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.276648 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:19:07.276745 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.276754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.276834 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.277005 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:07.277453 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.278204 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.279953 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.280587 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.280752 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.280816 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.281139 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.281219 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260812 11:19:07.281405 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.276168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37147 (local address 127.27.109.3:37475)
0812 11:19:07.276352 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:07.276373 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:07.276520 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:07.277045 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:07.277050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:07.277076 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:07.277261 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.277269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.278387 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.278391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.279760 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.279771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.280772 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.280775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.280929 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.280959 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:19:07.281096 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 11:19:07.281215 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":5}
I20260812 11:19:07.282285 21744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7d9e3441ad4e6a9f4644673979b4a4" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" is_pre_election: true
I20260812 11:19:07.282668 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.276566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39283 (local address 127.27.109.3:48413)
0812 11:19:07.277200 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:07.277219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:07.277413 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:07.277932 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:07.277936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:07.277958 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:07.278119 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.278126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.279495 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.279499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.280923 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:07.280935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.282123 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.282127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:07.282243 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.282278 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:19:07.282391 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:19:07.282490 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":38,"thread_start_us":234,"threads_started":1}
I20260812 11:19:07.282709 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.276849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48413 (local address 127.27.109.1:39283)
0812 11:19:07.277162 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.277167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.277333 (+ 166us) server_negotiation.cc:407] Connection header received
0812 11:19:07.277653 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.277656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.277739 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.277872 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:07.278284 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.279315 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.281100 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.281942 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.282068 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.282228 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.282430 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.282512 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
I20260812 11:19:07.282744 21744 raft_consensus.cc:2454] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 0.
I20260812 11:19:07.283495 21872 leader_election.cc:302] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0909d21c5a4ddeb08a5af61289dd75, a48cf6c690514b0080ce2b4b6d6db621; no voters:
I20260812 11:19:07.283509 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7d9e3441ad4e6a9f4644673979b4a4" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a61157fc09d491ca165ebd8ba700692" is_pre_election: true
I20260812 11:19:07.283859 21548 raft_consensus.cc:2454] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 0.
I20260812 11:19:07.283951 22092 raft_consensus.cc:2781] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:07.284068 22092 raft_consensus.cc:482] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:07.284138 22092 raft_consensus.cc:3037] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:07.285388 22092 raft_consensus.cc:504] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.285843 22092 leader_election.cc:288] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283)
I20260812 11:19:07.286486 21744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7d9e3441ad4e6a9f4644673979b4a4" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0909d21c5a4ddeb08a5af61289dd75"
I20260812 11:19:07.286648 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7d9e3441ad4e6a9f4644673979b4a4" candidate_uuid: "a48cf6c690514b0080ce2b4b6d6db621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a61157fc09d491ca165ebd8ba700692"
I20260812 11:19:07.286675 21744 raft_consensus.cc:3037] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:07.286778 21548 raft_consensus.cc:3037] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:07.288497 21548 raft_consensus.cc:2454] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 1.
I20260812 11:19:07.288543 21744 raft_consensus.cc:2454] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48cf6c690514b0080ce2b4b6d6db621 in term 1.
I20260812 11:19:07.288938 21872 leader_election.cc:302] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0909d21c5a4ddeb08a5af61289dd75, a48cf6c690514b0080ce2b4b6d6db621; no voters:
I20260812 11:19:07.289275 22092 raft_consensus.cc:2781] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:07.289784 22092 raft_consensus.cc:686] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 LEADER]: Becoming Leader. State: Replica: a48cf6c690514b0080ce2b4b6d6db621, State: Running, Role: LEADER
I20260812 11:19:07.290320 22092 consensus_queue.cc:237] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:07.295650 21324 catalog_manager.cc:5495] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 reported cstate change: term changed from 0 to 1, leader changed from <none> to a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "a48cf6c690514b0080ce2b4b6d6db621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } health_report { overall_health: UNKNOWN } } }
I20260812 11:19:07.328044 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.322006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.27.109.3:38731)
0812 11:19:07.322357 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:07.322371 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:07.322974 (+ 603us) server_negotiation.cc:407] Connection header received
0812 11:19:07.323148 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:07.323152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:07.323245 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:07.323385 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:07.323893 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.324465 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.325620 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:07.325897 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:07.326008 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:07.326475 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:07.327317 (+ 842us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:19:07.327451 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:07.327624 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 11:19:07.327805 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 11:19:07.337253 21744 raft_consensus.cc:1261] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Refusing update from remote peer a48cf6c690514b0080ce2b4b6d6db621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:07.337251 21548 raft_consensus.cc:1261] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Refusing update from remote peer a48cf6c690514b0080ce2b4b6d6db621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:07.338380 22092 consensus_queue.cc:1035] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:07.338681 22098 consensus_queue.cc:1035] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:07.347078 22101 mvcc.cc:205] Tried to move back new op lower bound from 7317641409881595904 to 7317641409705373696. Current Snapshot: MvccSnapshot[applied={T|T < 7317641409881595904}]
I20260812 11:19:07.350092 22103 mvcc.cc:205] Tried to move back new op lower bound from 7317641409881595904 to 7317641409705373696. Current Snapshot: MvccSnapshot[applied={T|T < 7317641409881595904}]
I20260812 11:19:07.350757 22102 mvcc.cc:205] Tried to move back new op lower bound from 7317641409881595904 to 7317641409705373696. Current Snapshot: MvccSnapshot[applied={T|T < 7317641409881595904}]
I20260812 11:19:07.363512 21324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:07.363777 21324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:07.367599 21324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Sending DeleteTablet for 3 replicas of tablet 2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:07.368880 21518 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7d9e3441ad4e6a9f4644673979b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:07 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45512
I20260812 11:19:07.368858 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7d9e3441ad4e6a9f4644673979b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:07 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36022
I20260812 11:19:07.369153 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 21391
I20260812 11:19:07.369455 21951 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7d9e3441ad4e6a9f4644673979b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:07 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:53280
I20260812 11:19:07.382269 22115 tablet_replica.cc:330] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: stopping tablet replica
I20260812 11:19:07.382869 22116 tablet_replica.cc:330] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: stopping tablet replica
I20260812 11:19:07.382987 22115 raft_consensus.cc:2229] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:07.383383 22116 raft_consensus.cc:2229] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:19:07.383576 22115 raft_consensus.cc:2258] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:07.383949 22116 raft_consensus.cc:2258] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:07.386358 22115 ts_tablet_manager.cc:1905] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:07.389376 22116 ts_tablet_manager.cc:1905] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:07.394322 22116 ts_tablet_manager.cc:1918] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:07.394714 22116 log.cc:1196] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:07.395423 22116 ts_tablet_manager.cc:1939] T 2d7d9e3441ad4e6a9f4644673979b4a4 P a48cf6c690514b0080ce2b4b6d6db621: Deleting consensus metadata
I20260812 11:19:07.395434 22115 ts_tablet_manager.cc:1918] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:07.395578 22115 log.cc:1196] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:07.395968 22115 ts_tablet_manager.cc:1939] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 8e0909d21c5a4ddeb08a5af61289dd75: Deleting consensus metadata
I20260812 11:19:07.397857 21306 catalog_manager.cc:4841] TS 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147): tablet 2d7d9e3441ad4e6a9f4644673979b4a4 (table test-table [id=509e307453c7459398a449d72db37b17]) successfully deleted
I20260812 11:19:07.398142 21306 catalog_manager.cc:4841] TS a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731): tablet 2d7d9e3441ad4e6a9f4644673979b4a4 (table test-table [id=509e307453c7459398a449d72db37b17]) successfully deleted
W20260812 11:19:07.403208 21306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:39283 (error 108)
I20260812 11:19:07.403420 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 21636
I20260812 11:19:07.405162 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:07.404635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39283 (local address 127.0.0.1:45514)
0812 11:19:07.404952 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:07.405029 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:39283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
W20260812 11:19:07.405366 21306 catalog_manager.cc:4584] TS 1a61157fc09d491ca165ebd8ba700692 (127.27.109.1:39283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d7d9e3441ad4e6a9f4644673979b4a4: Network error: Client connection negotiation failed: client connection to 127.27.109.1:39283: connect: Connection refused (error 111)
I20260812 11:19:07.434202 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 21841
I20260812 11:19:07.457890 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 21276
I20260812 11:19:27.494376 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39975
--webserver_interface=127.27.109.62
--webserver_port=44117
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:27.639490 22119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:27.639938 22119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:27.640031 22119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:27.646395 22119 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:19:27.646615 22119 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:19:27.646656 22119 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:19:27.646689 22119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:27.646741 22119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:27.646822 22119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:27.647166 22119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:27.647480 22119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:27.655486 22119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:27.656942 22119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:27.658214 22119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:27.671583 22126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:27.671656 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:27.671586 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:27.671633 22119 server_base.cc:1034] running on GCE node
I20260812 11:19:27.673173 22119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:27.674787 22119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:27.676614 22119 hybrid_clock.cc:648] HybridClock initialized: now 1786533567676552 us; error 85 us; skew 500 ppm
Aug 12 11:19:27 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:27.686712 22119 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:19:27.688462 22119 webserver.cc:460] Webserver started at http://127.27.109.62:44117/ using document root <none> and password file <none>
I20260812 11:19:27.689242 22119 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:27.689342 22119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:27.696710 22119 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260812 11:19:27.704228 22139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:27.707018 22119 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.003s
I20260812 11:19:27.707285 22119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0ec05130ca604b85b843215609acbb6e"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "63858352619433a77ad2355b922456d5"
server_key_iv: "6266024cda315920acd807398b5ea282"
server_key_version: "encryptionkey@0"
I20260812 11:19:27.708160 22119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:27.742655 22119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:27.747612 22119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:27.747897 22119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:27.758412 22119 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39975
I20260812 11:19:27.758455 22230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39975 every 8 connection(s)
I20260812 11:19:27.760106 22119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:27.766964 22231 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:19:27.768340 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 22119
I20260812 11:19:27.768980 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:39283
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:39975
--webserver_port=43509
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:19:27.770072 22231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Bootstrap starting.
I20260812 11:19:27.797606 22231 log.cc:824] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:27.815744 22231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:19:27.816458 22231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: Bootstrap complete.
I20260812 11:19:27.823482 22231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:27.824175 22231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec05130ca604b85b843215609acbb6e, State: Initialized, Role: FOLLOWER
I20260812 11:19:27.824955 22231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:27.825188 22231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:27.825279 22231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:27.825421 22231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:19:27.828604 22231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:27.829145 22231 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec05130ca604b85b843215609acbb6e; no voters:
I20260812 11:19:27.829638 22231 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 11:19:27.830010 22237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 3 FOLLOWER]: Leader election won for term 3
I20260812 11:19:27.830459 22237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [term 3 LEADER]: Becoming Leader. State: Replica: 0ec05130ca604b85b843215609acbb6e, State: Running, Role: LEADER
I20260812 11:19:27.831183 22231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:27.831094 22237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } }
I20260812 11:19:27.833408 22241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec05130ca604b85b843215609acbb6e. Latest consensus state: current_term: 3 leader_uuid: "0ec05130ca604b85b843215609acbb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } } }
I20260812 11:19:27.833629 22241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:27.834093 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0ec05130ca604b85b843215609acbb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec05130ca604b85b843215609acbb6e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39975 } } }
I20260812 11:19:27.834584 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:27.838649 22264 catalog_manager.cc:1259] Loaded cluster ID: bc16c2323d454397b029e6a1babfd3c7
I20260812 11:19:27.838775 22264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: loading cluster ID for follower catalog manager: success
I20260812 11:19:27.841709 22264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ec05130ca604b85b843215609acbb6e: acquiring CA information for follower catalog manager: success
F20260812 11:19:27.844971 22264 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffae0c619ad google::LogMessage::Fail() at ??:0
@ 0x7ffae0c6488a google::LogMessage::SendToLog() at ??:0
@ 0x7ffae0c614de google::LogMessage::Flush() at ??:0
@ 0x7ffae0c62a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffae392518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ffaeb5c6989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7ffaeb5a3023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7ffaeb5a354b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7ffaeb59c652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7ffaeb59b9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7ffaeb5e2541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ffae2ea4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffae44e86db start_thread at ??:0
@ 0x7ffadf75f61f clone at ??:0
W20260812 11:19:27.993772 22233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:27.994385 22233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:27.994481 22233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:28.000288 22233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:28.000427 22233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:28.000520 22233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:28.008368 22233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:39283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=43509
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:28.010048 22233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:28.011591 22233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:28.024701 22300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:28.024827 22306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:28.024762 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:28.025390 22233 server_base.cc:1034] running on GCE node
I20260812 11:19:28.026032 22233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:28.027066 22233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:28.030821 22233 hybrid_clock.cc:648] HybridClock initialized: now 1786533568030572 us; error 235 us; skew 500 ppm
Aug 12 11:19:28 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:28.035770 22233 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:19:28.037299 22233 webserver.cc:460] Webserver started at http://127.27.109.1:43509/ using document root <none> and password file <none>
I20260812 11:19:28.038183 22233 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:28.038295 22233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:28.045420 22233 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:19:28.049786 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:28.052266 22233 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:19:28.052587 22233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "1a61157fc09d491ca165ebd8ba700692"
format_stamp: "Formatted at 2026-08-12 11:18:44 on dist-test-slave-7z32"
server_key: "ecfe34bf6338f3b2e703a2be094b4051"
server_key_iv: "1cc3d66d93b8c159d33da0128597cbaf"
server_key_version: "encryptionkey@0"
I20260812 11:19:28.053572 22233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:28.075860 22233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:28.080737 22233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:28.081104 22233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:28.082036 22233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:28.083811 22330 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:19:28.089867 22233 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:19:28.089985 22233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260812 11:19:28.090061 22233 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:19:28.092813 22233 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:19:28.092916 22233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20260812 11:19:28.093277 22330 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap starting.
I20260812 11:19:28.110944 22233 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:39283
I20260812 11:19:28.111373 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:39283 every 8 connection(s)
I20260812 11:19:28.112897 22233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:28.114861 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 22233
I20260812 11:19:28.115510 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:37147
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:39975
--webserver_port=33725
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:19:28.125311 22330 log.cc:824] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:28.168259 22330 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:19:28.168987 22330 tablet_bootstrap.cc:492] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Bootstrap complete.
I20260812 11:19:28.171551 22330 ts_tablet_manager.cc:1397] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Time spent bootstrapping tablet: real 0.079s user 0.035s sys 0.017s
W20260812 11:19:28.174206 22527 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:19:28.117492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:36805)
0812 11:19:28.119943 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.120056 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:28.122136 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:28.172868 (+ 50732us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1770,"thread_start_us":281,"threads_started":1}
W20260812 11:19:28.175050 22521 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:39975 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:39975: Client connection negotiation failed: client connection to 127.27.109.62:39975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:19:28.176524 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.176140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:55069)
0812 11:19:28.176307 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.176418 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 11:19:28.177599 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.177333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:39387)
0812 11:19:28.177477 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.177538 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 11:19:28.177915 22521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:19:28.177436 22330 raft_consensus.cc:348] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:28.178352 22330 raft_consensus.cc:729] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a61157fc09d491ca165ebd8ba700692, State: Initialized, Role: FOLLOWER
I20260812 11:19:28.179231 22330 consensus_queue.cc:260] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:28.180984 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.180641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:32863)
0812 11:19:28.180838 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.180908 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 11:19:28.181457 22330 ts_tablet_manager.cc:1428] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260812 11:19:28.182710 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.182451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:34915)
0812 11:19:28.182582 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.182647 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 11:19:28.325335 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:28.325845 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:28.325906 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:28.331887 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:28.332021 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:28.332103 22526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:28.339785 22526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=33725
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:28.341658 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:28.343276 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:28.357862 22543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:28.357967 22544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:28.357971 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:28.358577 22526 server_base.cc:1034] running on GCE node
I20260812 11:19:28.359325 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:28.360512 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:28.361780 22526 hybrid_clock.cc:648] HybridClock initialized: now 1786533568361758 us; error 45 us; skew 500 ppm
Aug 12 11:19:28 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:28.366462 22526 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:19:28.368016 22526 webserver.cc:460] Webserver started at http://127.27.109.2:33725/ using document root <none> and password file <none>
I20260812 11:19:28.368906 22526 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:28.369009 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:28.375366 22526 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:19:28.379200 22558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:28.381464 22526 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:19:28.381712 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "8e0909d21c5a4ddeb08a5af61289dd75"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "7db6d9c978bb8b5464eb9cdd13be45dc"
server_key_iv: "483fd857ad325c86608ba0b11030eba7"
server_key_version: "encryptionkey@0"
I20260812 11:19:28.382732 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:28.412664 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:28.419114 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:28.419466 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:28.420902 22526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:28.422254 22526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:28.422322 22526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:28.422382 22526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:28.422399 22526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:28.437943 22526 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:37147
I20260812 11:19:28.437968 22750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:37147 every 8 connection(s)
I20260812 11:19:28.439715 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:28.442654 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 22526
I20260812 11:19:28.443361 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:38731
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:39975
--webserver_port=35387
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:19:28.445948 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.443242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:46547)
0812 11:19:28.444480 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.444705 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":230,"threads_started":1}
W20260812 11:19:28.446913 22751 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:39975 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:39975: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
I20260812 11:19:28.448004 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.447688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:33923)
0812 11:19:28.447853 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.447928 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 11:19:28.449115 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.448643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:46141)
0812 11:19:28.448873 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.448978 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 11:19:28.449472 22751 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:19:28.607569 22756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:28.608018 22756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:28.608076 22756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:28.612967 22756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:28.613088 22756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:28.613164 22756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:28.621668 22756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:38731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=35387
--tserver_master_addrs=127.27.109.62:39975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:28.623515 22756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:28.625067 22756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:19:28.639662 22756 server_base.cc:1034] running on GCE node
W20260812 11:19:28.639688 22765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:28.639698 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:28.639933 22766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:28.640569 22756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:28.641525 22756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:28.644994 22756 hybrid_clock.cc:648] HybridClock initialized: now 1786533568644887 us; error 95 us; skew 500 ppm
Aug 12 11:19:28 dist-test-slave-7z32 krb5kdc[20376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:28.649932 22756 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:19:28.651677 22756 webserver.cc:460] Webserver started at http://127.27.109.3:35387/ using document root <none> and password file <none>
I20260812 11:19:28.652573 22756 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:28.652688 22756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:28.659147 22756 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 11:19:28.663406 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:28.665786 22756 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:19:28.666082 22756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "a48cf6c690514b0080ce2b4b6d6db621"
format_stamp: "Formatted at 2026-08-12 11:18:45 on dist-test-slave-7z32"
server_key: "2b77b8b0c5a03f3f59b52a1717ae802a"
server_key_iv: "e7d96a3dda5793186964b091dd1c6879"
server_key_version: "encryptionkey@0"
I20260812 11:19:28.667086 22756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:28.698017 22756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:28.704392 22756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:28.704767 22756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:28.705770 22756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:28.707253 22756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:28.707347 22756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:28.707433 22756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:28.707479 22756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:28.726228 22756 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38731
I20260812 11:19:28.726596 22960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38731 every 8 connection(s)
I20260812 11:19:28.729408 22756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:28.736460 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.733310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:56937)
0812 11:19:28.734208 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.734885 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":297,"threads_started":1}
W20260812 11:19:28.737383 22961 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:39975 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:39975: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
I20260812 11:19:28.737704 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 22756
I20260812 11:19:28.739523 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.738745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:60699)
0812 11:19:28.739208 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.739397 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260812 11:19:28.740828 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:28.740423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:34743)
0812 11:19:28.740683 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:28.740752 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260812 11:19:28.741626 22961 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:19:29.184947 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:29.184046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:36041)
0812 11:19:29.184604 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:29.184772 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":136,"threads_started":1}
I20260812 11:19:29.451299 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:29.450487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:54611)
0812 11:19:29.451049 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:29.451146 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":115,"threads_started":1}
I20260812 11:19:29.629709 22971 raft_consensus.cc:482] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:29.630096 22971 raft_consensus.cc:504] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:29.631822 22971 leader_election.cc:288] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731)
Aug 12 11:19:29 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
Aug 12 11:19:29 dist-test-slave-7z32 krb5kdc[20376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260812 11:19:29.650601 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:29.632865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57413 (local address 127.27.109.3:38731)
0812 11:19:29.633287 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:29.633309 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:29.633383 (+ 74us) server_negotiation.cc:407] Connection header received
0812 11:19:29.634348 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:29.634357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:29.634779 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:29.635212 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:29.637025 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.638268 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.640087 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.640691 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.640912 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:29.644474 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:29.644481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:29.644495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:29.644532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:29.647399 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:29.648193 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:29.648201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:29.648205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:29.648355 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:29.648772 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:29.648776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:29.648778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:29.649299 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:29.649468 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:29.650096 (+ 628us) server_negotiation.cc:299] Negotiation successful
0812 11:19:29.650393 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260812 11:19:29.650589 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:29.632576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38731 (local address 127.27.109.1:57413)
0812 11:19:29.633170 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:29.633193 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:29.633396 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:29.635331 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:29.635341 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:29.636123 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:29.636779 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:29.636800 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.638443 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.638448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:29.639935 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:29.639948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.640851 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.640855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:29.641116 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:29.641986 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:29.642007 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:29.644164 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:29.647586 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:29.647595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:29.647615 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:29.648031 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:29.648481 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:29.648485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:29.648488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:29.648658 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:29.649438 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:29.649446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:29.649715 (+ 269us) client_negotiation.cc:769] Sending connection context
0812 11:19:29.649940 (+ 225us) client_negotiation.cc:240] Negotiation successful
0812 11:19:29.650381 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":143,"spinlock_wait_cycles":62592,"thread_start_us":167,"threads_started":1}
I20260812 11:19:29.651273 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:29.632192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37147 (local address 127.27.109.1:45109)
0812 11:19:29.632349 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:29.632369 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:29.632519 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:29.635329 (+ 2810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:29.635338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:29.636121 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:29.636780 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:29.636803 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.638373 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.638377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:29.639931 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:29.639945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.640872 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.640876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:29.641128 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:29.642003 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:29.642024 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:29.644504 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:29.647776 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:29.647780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:29.647783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:29.648227 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:29.648851 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:29.648855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:29.648857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:29.648984 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:29.650163 (+ 1179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:29.650167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:29.650322 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 11:19:29.650453 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 11:19:29.651037 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":186}
I20260812 11:19:29.651970 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:29.632549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:45109 (local address 127.27.109.2:37147)
0812 11:19:29.633195 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:29.633231 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:29.633295 (+ 64us) server_negotiation.cc:407] Connection header received
0812 11:19:29.634237 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:29.634246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:29.634758 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:29.635225 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:29.637025 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.638173 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.640122 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:29.640694 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:29.640916 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:29.644832 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:29.644840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:29.644860 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:29.644906 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:29.647505 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:29.648449 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:29.648457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:29.648465 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:29.648681 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:29.649116 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:29.649120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:29.649122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:29.649893 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:29.650090 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:29.651220 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0812 11:19:29.651742 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":248,"threads_started":1}
W20260812 11:19:29.654776 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
W20260812 11:19:29.655206 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:29.655298 22326 leader_election.cc:302] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a61157fc09d491ca165ebd8ba700692; no voters: 8e0909d21c5a4ddeb08a5af61289dd75, a48cf6c690514b0080ce2b4b6d6db621
I20260812 11:19:29.655700 22971 raft_consensus.cc:2726] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:19:29.743692 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:29.742765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:54591)
0812 11:19:29.743311 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:29.743431 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":213,"threads_started":1}
I20260812 11:19:30.187083 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:30.186231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:40785)
0812 11:19:30.186758 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:30.186918 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":113,"threads_started":1}
I20260812 11:19:30.453130 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:30.452423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:54979)
0812 11:19:30.452893 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:30.452974 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":114,"threads_started":1}
I20260812 11:19:30.745606 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:30.744889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:55617)
0812 11:19:30.745317 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:30.745417 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":123,"threads_started":1}
I20260812 11:19:31.189505 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:31.188656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:49577)
0812 11:19:31.189247 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:31.189358 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":136,"threads_started":1}
I20260812 11:19:31.454945 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:31.454243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:41577)
0812 11:19:31.454655 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:31.454736 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":122,"threads_started":1}
I20260812 11:19:31.511144 22987 raft_consensus.cc:482] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:31.511354 22987 raft_consensus.cc:504] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:31.511879 22987 leader_election.cc:288] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731)
W20260812 11:19:31.513365 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
W20260812 11:19:31.513607 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:31.513672 22326 leader_election.cc:302] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a61157fc09d491ca165ebd8ba700692; no voters: 8e0909d21c5a4ddeb08a5af61289dd75, a48cf6c690514b0080ce2b4b6d6db621
I20260812 11:19:31.513900 22987 raft_consensus.cc:2726] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:19:31.747603 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:31.746950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:56263)
0812 11:19:31.747379 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:31.747474 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260812 11:19:32.191885 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:32.191264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:35891)
0812 11:19:32.191676 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:32.191765 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260812 11:19:32.456702 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:32.456060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:56399)
0812 11:19:32.456427 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:32.456575 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260812 11:19:32.749560 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:32.748933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:33483)
0812 11:19:32.749368 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:32.749440 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":145,"threads_started":1}
I20260812 11:19:33.194054 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:33.193361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:35005)
0812 11:19:33.193783 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:33.193880 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":170,"threads_started":1}
I20260812 11:19:33.458667 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:33.457998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:51007)
0812 11:19:33.458444 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:33.458540 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260812 11:19:33.599306 22999 raft_consensus.cc:482] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:33.599538 22999 raft_consensus.cc:504] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:33.600109 22999 leader_election.cc:288] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731)
W20260812 11:19:33.601430 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
W20260812 11:19:33.601621 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:33.601682 22326 leader_election.cc:302] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a61157fc09d491ca165ebd8ba700692; no voters: 8e0909d21c5a4ddeb08a5af61289dd75, a48cf6c690514b0080ce2b4b6d6db621
I20260812 11:19:33.601943 22999 raft_consensus.cc:2726] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:19:33.751401 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:33.750667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:36299)
0812 11:19:33.751191 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:33.751282 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":244,"threads_started":1}
I20260812 11:19:34.196450 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:34.195769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:52411)
0812 11:19:34.196229 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:34.196321 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":146,"threads_started":1}
I20260812 11:19:34.461094 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:34.460314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:60227)
0812 11:19:34.460870 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:34.460969 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260812 11:19:34.753612 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:34.752789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:54383)
0812 11:19:34.753379 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:34.753484 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":160,"threads_started":1}
I20260812 11:19:35.198523 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:35.197597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:53525)
0812 11:19:35.198126 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:35.198373 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":182,"threads_started":1}
I20260812 11:19:35.462968 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:35.462210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:35707)
0812 11:19:35.462675 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:35.462764 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":130,"threads_started":1}
I20260812 11:19:35.755772 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:35.755038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:46247)
0812 11:19:35.755494 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:35.755593 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":95,"threads_started":1}
I20260812 11:19:36.200690 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:36.200001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:60919)
0812 11:19:36.200425 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:36.200533 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":118,"threads_started":1}
I20260812 11:19:36.464897 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:36.464232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:43525)
0812 11:19:36.464646 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:36.464749 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":114,"threads_started":1}
I20260812 11:19:36.757795 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:36.757072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:60995)
0812 11:19:36.757564 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:36.757662 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":125,"threads_started":1}
I20260812 11:19:37.202425 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:37.201850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:47153)
0812 11:19:37.202232 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:37.202312 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":140,"threads_started":1}
I20260812 11:19:37.467061 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:37.466328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:36655)
0812 11:19:37.466778 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:37.466923 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":156,"threads_started":1}
I20260812 11:19:37.759721 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:37.759091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:50907)
0812 11:19:37.759464 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:37.759540 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":169,"threads_started":1}
I20260812 11:19:38.204715 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:38.203955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:43501)
0812 11:19:38.204403 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:38.204505 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":118,"threads_started":1}
I20260812 11:19:38.214053 23024 raft_consensus.cc:482] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:38.214283 23024 raft_consensus.cc:504] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:38.215183 23024 leader_election.cc:288] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731)
W20260812 11:19:38.216619 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
W20260812 11:19:38.216862 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:38.216945 22326 leader_election.cc:302] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a61157fc09d491ca165ebd8ba700692; no voters: 8e0909d21c5a4ddeb08a5af61289dd75, a48cf6c690514b0080ce2b4b6d6db621
I20260812 11:19:38.217283 23024 raft_consensus.cc:2726] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:19:38.469034 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:38.468338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:40277)
0812 11:19:38.468798 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:38.468895 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":125,"threads_started":1}
I20260812 11:19:38.762059 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:38.761173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:60811)
0812 11:19:38.761637 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:38.761826 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":125,"threads_started":1}
I20260812 11:19:39.206686 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:39.205920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:34485)
0812 11:19:39.206399 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:39.206487 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":136,"threads_started":1}
I20260812 11:19:39.471619 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:39.470511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:55749)
0812 11:19:39.471202 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:39.471336 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":164,"threads_started":1}
I20260812 11:19:39.763926 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:39.763131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:47529)
0812 11:19:39.763621 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:39.763777 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":112,"threads_started":1}
I20260812 11:19:40.208792 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:40.208073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:48867)
0812 11:19:40.208575 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:40.208668 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":124,"threads_started":1}
I20260812 11:19:40.473539 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:40.472904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:54085)
0812 11:19:40.473325 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:40.473414 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260812 11:19:40.765796 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:40.764990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:53181)
0812 11:19:40.765508 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:40.765631 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":98,"threads_started":1}
I20260812 11:19:41.210918 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:41.210211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:35097)
0812 11:19:41.210620 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:41.210707 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":169,"threads_started":1}
I20260812 11:19:41.475745 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:41.475063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:33327)
0812 11:19:41.475545 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:41.475628 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":153,"threads_started":1}
I20260812 11:19:41.767834 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:41.767225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:48009)
0812 11:19:41.767633 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:41.767707 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":118,"threads_started":1}
I20260812 11:19:42.213348 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:42.212415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:60509)
0812 11:19:42.213082 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:42.213190 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":242,"threads_started":1}
I20260812 11:19:42.239104 23041 raft_consensus.cc:482] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:42.239326 23041 raft_consensus.cc:504] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0909d21c5a4ddeb08a5af61289dd75" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37147 } } peers { permanent_uuid: "a48cf6c690514b0080ce2b4b6d6db621" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38731 } } peers { permanent_uuid: "1a61157fc09d491ca165ebd8ba700692" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39283 } }
I20260812 11:19:42.240106 23041 leader_election.cc:288] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147), a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731)
W20260812 11:19:42.241345 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a48cf6c690514b0080ce2b4b6d6db621 (127.27.109.3:38731): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
W20260812 11:19:42.241590 22326 leader_election.cc:341] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e0909d21c5a4ddeb08a5af61289dd75 (127.27.109.2:37147): Not found: Tablet not found: 2d7d9e3441ad4e6a9f4644673979b4a4
I20260812 11:19:42.241792 22326 leader_election.cc:302] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a61157fc09d491ca165ebd8ba700692; no voters: 8e0909d21c5a4ddeb08a5af61289dd75, a48cf6c690514b0080ce2b4b6d6db621
I20260812 11:19:42.242130 23041 raft_consensus.cc:2726] T 2d7d9e3441ad4e6a9f4644673979b4a4 P 1a61157fc09d491ca165ebd8ba700692 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:19:42.477720 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:42.476993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:45097)
0812 11:19:42.477514 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:42.477607 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":156,"threads_started":1}
I20260812 11:19:42.769819 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:42.769152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:50531)
0812 11:19:42.769577 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:42.769665 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":131,"threads_started":1}
I20260812 11:19:43.215296 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:43.214730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.1:51275)
0812 11:19:43.215114 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:43.215188 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":200,"threads_started":1}
I20260812 11:19:43.480189 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:43.479432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.2:53655)
0812 11:19:43.479917 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:43.480018 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":109,"threads_started":1}
I20260812 11:19:43.750221 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 22233
I20260812 11:19:43.780565 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:43.779866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39975 (local address 127.27.109.3:50169)
0812 11:19:43.780244 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:43.780338 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":147,"threads_started":1}
I20260812 11:19:43.787012 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 22526
I20260812 11:19:43.807286 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 22756
2026-08-12T11:19:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (61330 ms)
[----------] 30 tests from SecurityITest (300994 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:19:43 dist-test-slave-7z32 krb5kdc[23052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:19:43 dist-test-slave-7z32 krb5kdc[23052](info): set up 2 sockets
Aug 12 11:19:43 dist-test-slave-7z32 krb5kdc[23052](info): commencing operation
krb5kdc: starting...
W20260812 11:19:45.277603 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.434s user 0.010s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:19:45 dist-test-slave-7z32 krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:19:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:19:45.456422 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39125
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:45.616890 23071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:45.617339 23071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:45.617390 23071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:45.622254 23071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:45.622385 23071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:45.622408 23071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:45.622428 23071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:45.622445 23071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:45.629679 23071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:45.631534 23071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:45.633042 23071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:45.648195 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:45.648313 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:45.648748 23083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:45.648275 23071 server_base.cc:1034] running on GCE node
I20260812 11:19:45.649772 23071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:45.651782 23071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:45.655306 23071 hybrid_clock.cc:648] HybridClock initialized: now 1786533585655175 us; error 93 us; skew 500 ppm
Aug 12 11:19:45 dist-test-slave-7z32 krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533585, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:45.661018 23071 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:19:45.663247 23071 webserver.cc:460] Webserver started at http://127.27.109.62:41473/ using document root <none> and password file <none>
I20260812 11:19:45.664276 23071 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:45.664394 23071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:45.664763 23071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:45.668249 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "c0cb589e4d364816b1704c3945caccc8"
format_stamp: "Formatted at 2026-08-12 11:19:45 on dist-test-slave-7z32"
server_key: "6f2c90e867a908817671a0f708fe26ba"
server_key_iv: "01da9898ea59a92b504a4369d5ae0117"
server_key_version: "encryptionkey@0"
I20260812 11:19:45.669222 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "c0cb589e4d364816b1704c3945caccc8"
format_stamp: "Formatted at 2026-08-12 11:19:45 on dist-test-slave-7z32"
server_key: "6f2c90e867a908817671a0f708fe26ba"
server_key_iv: "01da9898ea59a92b504a4369d5ae0117"
server_key_version: "encryptionkey@0"
I20260812 11:19:45.675621 23071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:19:45.679795 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:45.682255 23071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:19:45.682549 23071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "c0cb589e4d364816b1704c3945caccc8"
format_stamp: "Formatted at 2026-08-12 11:19:45 on dist-test-slave-7z32"
server_key: "6f2c90e867a908817671a0f708fe26ba"
server_key_iv: "01da9898ea59a92b504a4369d5ae0117"
server_key_version: "encryptionkey@0"
I20260812 11:19:45.682772 23071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:45.711835 23071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:45.717551 23071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:45.718035 23071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:45.731000 23071 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39125
I20260812 11:19:45.731106 23188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39125 every 8 connection(s)
I20260812 11:19:45.732615 23071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:45.736976 23190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:45.740424 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 23071
I20260812 11:19:45.740658 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:19:45.741097 28084 external_mini_cluster.cc:1428] Setting key 4506bac24d8322ab5c5b8add22d40c90
I20260812 11:19:45.745405 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8: Bootstrap starting.
I20260812 11:19:45.748974 23190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:45.749979 23190 log.cc:824] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:45.753401 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8: No bootstrap required, opened a new log
Aug 12 11:19:45 dist-test-slave-7z32 krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:45.758463 23190 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb589e4d364816b1704c3945caccc8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39125 } }
I20260812 11:19:45.758805 23190 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:45.758977 23190 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cb589e4d364816b1704c3945caccc8, State: Initialized, Role: FOLLOWER
I20260812 11:19:45.759497 23190 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb589e4d364816b1704c3945caccc8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39125 } }
I20260812 11:19:45.759676 23190 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:45.759769 23190 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:45.759883 23190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:45.761087 23190 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb589e4d364816b1704c3945caccc8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39125 } }
I20260812 11:19:45.761518 23190 leader_election.cc:302] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0cb589e4d364816b1704c3945caccc8; no voters:
I20260812 11:19:45.761942 23190 leader_election.cc:288] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:19:45.762053 23197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:45.762346 23197 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [term 1 LEADER]: Becoming Leader. State: Replica: c0cb589e4d364816b1704c3945caccc8, State: Running, Role: LEADER
I20260812 11:19:45.762782 23197 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb589e4d364816b1704c3945caccc8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39125 } }
I20260812 11:19:45.763697 23190 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:45.764376 23197 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0cb589e4d364816b1704c3945caccc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb589e4d364816b1704c3945caccc8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39125 } } }
I20260812 11:19:45.764654 23197 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:45.764895 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0cb589e4d364816b1704c3945caccc8. Latest consensus state: current_term: 1 leader_uuid: "c0cb589e4d364816b1704c3945caccc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb589e4d364816b1704c3945caccc8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39125 } } }
I20260812 11:19:45.765071 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:45.765198 23216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:19:45.765231 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:45.743760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.27.109.62:39125)
0812 11:19:45.744570 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:45.744593 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:45.744654 (+ 61us) server_negotiation.cc:407] Connection header received
0812 11:19:45.745687 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:45.745764 (+ 77us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:45.746352 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:45.746802 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:45.748173 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:45.749734 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:45.751198 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:45.751926 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:45.752162 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:45.756629 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:45.756639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:45.756667 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:45.756722 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:45.759507 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:45.760344 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:45.760352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:45.760360 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:45.760540 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:45.760920 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:45.760930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:45.760932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:45.761894 (+ 962us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:45.762243 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:45.763251 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0812 11:19:45.763725 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":328,"threads_started":1}
I20260812 11:19:45.769682 23216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:19:45.779373 23216 catalog_manager.cc:1347] Generated new cluster ID: 37036579f553419eb356725b5eeb6a0d
I20260812 11:19:45.779517 23216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:19:45.803256 23216 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:19:45.804812 23216 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:19:45.814965 23216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8: Generated new TSK 0
I20260812 11:19:45.816210 23216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:19:45.877288 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39125
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:46.050007 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:46.050539 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:46.050663 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:46.056541 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:46.056843 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:46.057017 23240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:46.064365 23240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:46.066062 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:46.067652 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:46.081682 23252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:46.081954 23255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:46.082206 23240 server_base.cc:1034] running on GCE node
W20260812 11:19:46.081682 23250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:46.083099 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:46.083971 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:46.085201 23240 hybrid_clock.cc:648] HybridClock initialized: now 1786533586085156 us; error 76 us; skew 500 ppm
Aug 12 11:19:46 dist-test-slave-7z32 krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533586, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:46.089968 23240 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:19:46.091346 23240 webserver.cc:460] Webserver started at http://127.27.109.1:42381/ using document root <none> and password file <none>
I20260812 11:19:46.092329 23240 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:46.092418 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:46.092782 23240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:46.095329 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "8ea5f000e80a4eb8b1ccab8929817603"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "120b94be364af22ad97e7df0593a2e86"
server_key_iv: "cc3b4cc582a14a83b6b31adb0f8bc64a"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.096022 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "8ea5f000e80a4eb8b1ccab8929817603"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "120b94be364af22ad97e7df0593a2e86"
server_key_iv: "cc3b4cc582a14a83b6b31adb0f8bc64a"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.101647 23240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:19:46.105444 23269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.107501 23240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 11:19:46.107710 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "8ea5f000e80a4eb8b1ccab8929817603"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "120b94be364af22ad97e7df0593a2e86"
server_key_iv: "cc3b4cc582a14a83b6b31adb0f8bc64a"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.107870 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:46.128854 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:46.132637 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:46.132892 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:46.133605 23240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:46.134598 23240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:46.134678 23240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.134750 23240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:46.134804 23240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.147147 23240 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:44095
I20260812 11:19:46.147190 23468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:44095 every 8 connection(s)
I20260812 11:19:46.148382 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:46.151194 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 23240
I20260812 11:19:46.151350 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:19:46.151698 28084 external_mini_cluster.cc:1428] Setting key 3821be941c60d800f35457da731004ac
Aug 12 11:19:46 dist-test-slave-7z32 krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533586, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:19:46.168567 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.151315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:59741 (local address 127.27.109.62:39125)
0812 11:19:46.151507 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.151514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.153035 (+ 1521us) server_negotiation.cc:407] Connection header received
0812 11:19:46.154043 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.154048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.154122 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.154241 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:46.155647 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.156604 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.158270 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.158567 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.158711 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.162651 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:46.162656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:46.162658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:46.162691 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:46.165266 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.165945 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.165950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.165952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.166127 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.166541 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.166546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.166548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.166873 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:46.166987 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.168128 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.168371 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:19:46.169327 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.151852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39125 (local address 127.27.109.1:59741)
0812 11:19:46.152819 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.152883 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.153778 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.154444 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.154451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.154875 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:46.155417 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.155430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.156905 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.156910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.158068 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.158082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.158788 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.158793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.159064 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.160009 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:46.160048 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:46.162325 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:46.165431 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:46.165439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:46.165463 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:46.165808 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:46.166241 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:46.166244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:46.166246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:46.166427 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:46.167018 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:46.167028 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:46.167552 (+ 524us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.167873 (+ 321us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.168355 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":185,"threads_started":1}
I20260812 11:19:46.171196 23471 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39125
I20260812 11:19:46.171595 23471 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:46.172402 23471 heartbeater.cc:511] Master 127.27.109.62:39125 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:19:46.175124 23123 ts_manager.cc:194] Registered new tserver with Master: 8ea5f000e80a4eb8b1ccab8929817603 (127.27.109.1:44095)
I20260812 11:19:46.176697 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:59741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:19:46.201125 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39125
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:46.369411 23481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:46.369814 23481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:46.369871 23481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:46.375216 23481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:46.375324 23481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:46.375397 23481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:46.381744 23481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:46.383745 23481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:46.385054 23481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:46.400171 23498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:46.400270 23497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:46.400300 23501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:46.401273 23481 server_base.cc:1034] running on GCE node
I20260812 11:19:46.402021 23481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:46.403087 23481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:46.406950 23481 hybrid_clock.cc:648] HybridClock initialized: now 1786533586406885 us; error 53 us; skew 500 ppm
Aug 12 11:19:46 dist-test-slave-7z32 krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533586, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:46.412550 23481 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:19:46.414453 23481 webserver.cc:460] Webserver started at http://127.27.109.2:46811/ using document root <none> and password file <none>
I20260812 11:19:46.415768 23481 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:46.415899 23481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:46.416285 23481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:46.420548 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "5384228089e74241ba690c1190656dd7"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "cfff0cef3a30c9456e2f48b5125283a3"
server_key_iv: "21799f10900ce78440c873d2fb312c6c"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.422506 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "5384228089e74241ba690c1190656dd7"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "cfff0cef3a30c9456e2f48b5125283a3"
server_key_iv: "21799f10900ce78440c873d2fb312c6c"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.429409 23481 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:19:46.434139 23513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.437212 23481 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:19:46.437537 23481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "5384228089e74241ba690c1190656dd7"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "cfff0cef3a30c9456e2f48b5125283a3"
server_key_iv: "21799f10900ce78440c873d2fb312c6c"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.437911 23481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:46.463850 23481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:46.469152 23481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:46.469513 23481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:46.470535 23481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:46.472260 23481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:46.472375 23481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.472463 23481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:46.472530 23481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.488174 23481 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45103
I20260812 11:19:46.488236 23713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45103 every 8 connection(s)
I20260812 11:19:46.489624 23481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:46.490948 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 23481
I20260812 11:19:46.491107 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:19:46.491506 28084 external_mini_cluster.cc:1428] Setting key e5d526c5101ae36f4405629f3878a989
Aug 12 11:19:46 dist-test-slave-7z32 krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533586, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:19:46.514017 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.492684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34681 (local address 127.27.109.62:39125)
0812 11:19:46.492913 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.492918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.494785 (+ 1867us) server_negotiation.cc:407] Connection header received
0812 11:19:46.496733 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.496741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.496843 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.497000 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:46.499654 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.500538 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.502243 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.502503 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.502636 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.506748 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:46.506752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:46.506754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:46.506787 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:46.509083 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.510207 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.510212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.510213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.510347 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.511090 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.511093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.511094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.511408 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:46.511702 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.513511 (+ 1809us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.513738 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:19:46.514549 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.493644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39125 (local address 127.27.109.2:34681)
0812 11:19:46.494516 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.494596 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.496374 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.497344 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.497357 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.498143 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:46.499314 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.499341 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.500894 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.500900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.502037 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.502054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.502707 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.502711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.502991 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.503965 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:46.503987 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:46.506471 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:46.509416 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:46.509429 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:46.509455 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:46.509991 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:46.510511 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:46.510514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:46.510516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:46.510910 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:46.511820 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:46.511833 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:46.512418 (+ 585us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.512758 (+ 340us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.513193 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":250,"threads_started":1}
I20260812 11:19:46.516330 23715 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39125
I20260812 11:19:46.516798 23715 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:46.517527 23715 heartbeater.cc:511] Master 127.27.109.62:39125 requested a full tablet report, sending...
I20260812 11:19:46.519698 23123 ts_manager.cc:194] Registered new tserver with Master: 5384228089e74241ba690c1190656dd7 (127.27.109.2:45103)
I20260812 11:19:46.520422 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:34681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:19:46.540819 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39125
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:46.716117 23723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:46.716604 23723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:46.716668 23723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:46.723896 23723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:46.724293 23723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:46.724749 23723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:46.732867 23723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:46.734558 23723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:46.736743 23723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:46.753463 23734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:46.753700 23738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:46.753746 23733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:46.753966 23723 server_base.cc:1034] running on GCE node
I20260812 11:19:46.755131 23723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:46.756186 23723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:46.759704 23723 hybrid_clock.cc:648] HybridClock initialized: now 1786533586759477 us; error 209 us; skew 500 ppm
Aug 12 11:19:46 dist-test-slave-7z32 krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533586, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:46.768509 23723 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:19:46.770195 23723 webserver.cc:460] Webserver started at http://127.27.109.3:37891/ using document root <none> and password file <none>
I20260812 11:19:46.771302 23723 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:46.771423 23723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:46.771687 23723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:46.775545 23723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "2f090ac440334007add53d66f47bd85d"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "09338791f1f07a2ea5227b336e400419"
server_key_iv: "9e41940d4334c323310870aebf91e147"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.776572 23723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "2f090ac440334007add53d66f47bd85d"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "09338791f1f07a2ea5227b336e400419"
server_key_iv: "9e41940d4334c323310870aebf91e147"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.783340 23723 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 11:19:46.788337 23747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.790964 23723 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:19:46.791220 23723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "2f090ac440334007add53d66f47bd85d"
format_stamp: "Formatted at 2026-08-12 11:19:46 on dist-test-slave-7z32"
server_key: "09338791f1f07a2ea5227b336e400419"
server_key_iv: "9e41940d4334c323310870aebf91e147"
server_key_version: "encryptionkey@0"
I20260812 11:19:46.791422 23723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:46.813591 23723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:46.818547 23723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:46.818969 23723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:46.819900 23723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:46.821280 23723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:46.821358 23723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.821439 23723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:46.821480 23723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:46.840133 23723 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:41671
I20260812 11:19:46.840157 23948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:41671 every 8 connection(s)
I20260812 11:19:46.841715 23723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:46.845930 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 23723
I20260812 11:19:46.846134 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:19:46.846721 28084 external_mini_cluster.cc:1428] Setting key 2319adbbdbda50048f085119446a2e33
Aug 12 11:19:46 dist-test-slave-7z32 krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533586, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:46.864310 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.844774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:59259 (local address 127.27.109.62:39125)
0812 11:19:46.845100 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.845220 (+ 120us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.847811 (+ 2591us) server_negotiation.cc:407] Connection header received
0812 11:19:46.849554 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.849559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.849761 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.849895 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:46.851758 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.852464 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.853975 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.854289 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.854545 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.858176 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:46.858191 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:46.858194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:46.858246 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:46.860622 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.861456 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.861460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.861462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.861683 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.862295 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.862302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.862307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.862653 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:46.862762 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.863888 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.864092 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260812 11:19:46.866160 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.845903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39125 (local address 127.27.109.3:59259)
0812 11:19:46.847443 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.847562 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.849069 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.850210 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.850222 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.850750 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:46.851523 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.851540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.852657 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.852660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.853734 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.853747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.854551 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.854555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.854748 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.855609 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:46.855628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:46.857834 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:46.860842 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:46.860858 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:46.860882 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:46.861311 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:46.861842 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:46.861845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:46.861847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:46.862155 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:46.862883 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:46.862894 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:46.863420 (+ 526us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.863825 (+ 405us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.864454 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":247,"threads_started":1}
I20260812 11:19:46.867810 23949 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39125
I20260812 11:19:46.868170 23949 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:46.868892 23949 heartbeater.cc:511] Master 127.27.109.62:39125 requested a full tablet report, sending...
I20260812 11:19:46.870409 23123 ts_manager.cc:194] Registered new tserver with Master: 2f090ac440334007add53d66f47bd85d (127.27.109.3:41671)
I20260812 11:19:46.871188 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:59259
I20260812 11:19:46.877763 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:19:46.890275 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.880623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.27.109.62:39125)
0812 11:19:46.880776 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.880780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.881117 (+ 337us) server_negotiation.cc:407] Connection header received
0812 11:19:46.881374 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.881378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.881429 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.881532 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:46.882417 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.883078 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.883994 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.884236 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.884377 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.886219 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:46.886223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:46.886225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:46.886255 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:46.888167 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.888702 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.888708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.888710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.888807 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:46.889173 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:46.889178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:46.889180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:46.889470 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:46.889637 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.889926 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.890070 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:19:46.894894 23123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:19:46.898892 23123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:19:46.923100 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.915293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.27.109.3:41671)
0812 11:19:46.916370 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.916396 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.916428 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:19:46.916547 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.916554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.916839 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.917084 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:46.917725 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.918923 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.920689 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.921587 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.921780 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.921997 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.922600 (+ 603us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.922813 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":282,"threads_started":1}
I20260812 11:19:46.923316 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.914010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45103 (local address 127.0.0.1:33764)
0812 11:19:46.914770 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.914879 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.915080 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.916978 (+ 1898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.916989 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.917023 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:46.917253 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.917264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.919504 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.919509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.920761 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.920780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.922635 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.922640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.922778 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.922814 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.922916 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.923034 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":18,"thread_start_us":224,"threads_started":1}
I20260812 11:19:46.922973 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.914615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41671 (local address 127.0.0.1:50794)
0812 11:19:46.915734 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.915751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.915869 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.917257 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.917260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.917272 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:46.917516 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.917526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.919102 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.919106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.920483 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.920497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.921789 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.921793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.922074 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.922280 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.922491 (+ 211us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.922709 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":144,"thread_start_us":120,"threads_started":1}
I20260812 11:19:46.923961 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.914233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.27.109.2:45103)
0812 11:19:46.916225 (+ 1992us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.916245 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.916282 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:19:46.916397 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.916406 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.916724 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.916939 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:46.917561 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.919274 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.921012 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.922383 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.923214 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.923483 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.923563 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.923743 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1832,"thread_start_us":268,"threads_started":1}
I20260812 11:19:46.926532 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.915475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44095 (local address 127.0.0.1:50854)
0812 11:19:46.916451 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.916471 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.916605 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.917237 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.917243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.917266 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:46.917522 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.917527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.919504 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.919509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.921977 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.922002 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.925843 (+ 3841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.925849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.926010 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.926052 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.926207 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.926334 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":63,"thread_start_us":188,"threads_started":1}
I20260812 11:19:46.926615 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.916083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.27.109.1:44095)
0812 11:19:46.916559 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.916565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.916590 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:19:46.916757 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.916762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.916943 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.917088 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:46.917700 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.919276 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.922286 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.925541 (+ 3255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.925703 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.925904 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.926280 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.926423 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":156,"threads_started":1}
I20260812 11:19:46.928579 23604 tablet_service.cc:1467] Processing CreateTablet for tablet 69dac5497a4e4b44bf67491c912b7c8b (DEFAULT_TABLE table=test-table [id=c44f5a561b9249ef9eb93847c1fa517c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:46.928740 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 69dac5497a4e4b44bf67491c912b7c8b (DEFAULT_TABLE table=test-table [id=c44f5a561b9249ef9eb93847c1fa517c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:46.930080 23604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69dac5497a4e4b44bf67491c912b7c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:46.930287 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69dac5497a4e4b44bf67491c912b7c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:46.931084 23361 tablet_service.cc:1467] Processing CreateTablet for tablet 69dac5497a4e4b44bf67491c912b7c8b (DEFAULT_TABLE table=test-table [id=c44f5a561b9249ef9eb93847c1fa517c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:46.932528 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69dac5497a4e4b44bf67491c912b7c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:46.938581 23976 tablet_bootstrap.cc:492] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Bootstrap starting.
I20260812 11:19:46.941442 23977 tablet_bootstrap.cc:492] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Bootstrap starting.
I20260812 11:19:46.942881 23976 tablet_bootstrap.cc:654] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:46.944147 23979 tablet_bootstrap.cc:492] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603: Bootstrap starting.
I20260812 11:19:46.944288 23977 tablet_bootstrap.cc:654] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:46.944669 23976 log.cc:824] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:46.945775 23977 log.cc:824] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:46.947345 23979 tablet_bootstrap.cc:654] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:46.948124 23976 tablet_bootstrap.cc:492] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: No bootstrap required, opened a new log
I20260812 11:19:46.948434 23976 ts_tablet_manager.cc:1397] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260812 11:19:46.948666 23979 log.cc:824] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:46.948892 23977 tablet_bootstrap.cc:492] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: No bootstrap required, opened a new log
I20260812 11:19:46.949151 23977 ts_tablet_manager.cc:1397] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:19:46.951993 23979 tablet_bootstrap.cc:492] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603: No bootstrap required, opened a new log
I20260812 11:19:46.952293 23979 ts_tablet_manager.cc:1397] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:19:46.953351 23977 raft_consensus.cc:348] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.953750 23977 raft_consensus.cc:374] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:46.953846 23977 raft_consensus.cc:729] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f090ac440334007add53d66f47bd85d, State: Initialized, Role: FOLLOWER
I20260812 11:19:46.953531 23976 raft_consensus.cc:348] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.953966 23976 raft_consensus.cc:374] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:46.954049 23976 raft_consensus.cc:729] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5384228089e74241ba690c1190656dd7, State: Initialized, Role: FOLLOWER
I20260812 11:19:46.954500 23977 consensus_queue.cc:260] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.954705 23976 consensus_queue.cc:260] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.955657 23949 heartbeater.cc:503] Master 127.27.109.62:39125 was elected leader, sending a full tablet report...
I20260812 11:19:46.955927 23977 ts_tablet_manager.cc:1428] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260812 11:19:46.956210 23976 ts_tablet_manager.cc:1428] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 11:19:46.956615 23715 heartbeater.cc:503] Master 127.27.109.62:39125 was elected leader, sending a full tablet report...
I20260812 11:19:46.958406 23979 raft_consensus.cc:348] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.958750 23979 raft_consensus.cc:374] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:46.958798 23979 raft_consensus.cc:729] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea5f000e80a4eb8b1ccab8929817603, State: Initialized, Role: FOLLOWER
I20260812 11:19:46.959022 23987 raft_consensus.cc:482] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:46.959270 23987 raft_consensus.cc:504] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.959410 23979 consensus_queue.cc:260] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.960781 23979 ts_tablet_manager.cc:1428] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 11:19:46.961735 23471 heartbeater.cc:503] Master 127.27.109.62:39125 was elected leader, sending a full tablet report...
I20260812 11:19:46.963238 23987 leader_election.cc:288] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ea5f000e80a4eb8b1ccab8929817603 (127.27.109.1:44095), 2f090ac440334007add53d66f47bd85d (127.27.109.3:41671)
I20260812 11:19:46.967370 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.961344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44095 (local address 127.27.109.2:34953)
0812 11:19:46.961596 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.961616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.961738 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.962609 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.962613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.962636 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:46.962867 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.962877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.964070 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.964074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.965538 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.965552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.966814 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.966818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.966968 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.966999 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.967088 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.967179 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 11:19:46.968025 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.962073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34953 (local address 127.27.109.1:44095)
0812 11:19:46.962215 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.962220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.962238 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:19:46.962298 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.962301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.962354 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.962441 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:46.963063 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.963884 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.965748 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.966512 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.966650 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.966720 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.967741 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.967865 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:19:46.968808 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dac5497a4e4b44bf67491c912b7c8b" candidate_uuid: "5384228089e74241ba690c1190656dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea5f000e80a4eb8b1ccab8929817603" is_pre_election: true
I20260812 11:19:46.969262 23394 raft_consensus.cc:2454] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5384228089e74241ba690c1190656dd7 in term 0.
I20260812 11:19:46.969499 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.963590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41671 (local address 127.27.109.2:58735)
0812 11:19:46.964175 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:46.964195 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:46.964306 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:46.964751 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:46.964754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:46.964779 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:46.964930 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.964937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.966264 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.966268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.967903 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:46.967915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.968988 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.968993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:46.969110 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.969141 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:19:46.969253 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 11:19:46.969350 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":191,"threads_started":1}
I20260812 11:19:46.969839 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:46.964291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:58735 (local address 127.27.109.3:41671)
0812 11:19:46.964445 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:46.964451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:46.964469 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:19:46.964539 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:46.964544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:46.964601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:46.964696 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:46.965196 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.966036 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.968138 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:46.968793 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:46.968936 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:46.968999 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:46.969400 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 11:19:46.969504 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:19:46.970181 23516 leader_election.cc:302] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5384228089e74241ba690c1190656dd7, 8ea5f000e80a4eb8b1ccab8929817603; no voters:
I20260812 11:19:46.970642 23987 raft_consensus.cc:2781] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:46.970722 23861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dac5497a4e4b44bf67491c912b7c8b" candidate_uuid: "5384228089e74241ba690c1190656dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f090ac440334007add53d66f47bd85d" is_pre_election: true
I20260812 11:19:46.970811 23987 raft_consensus.cc:482] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:46.970997 23987 raft_consensus.cc:3037] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:46.971186 23861 raft_consensus.cc:2454] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5384228089e74241ba690c1190656dd7 in term 0.
I20260812 11:19:46.972563 23987 raft_consensus.cc:504] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.973014 23987 leader_election.cc:288] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [CANDIDATE]: Term 1 election: Requested vote from peers 8ea5f000e80a4eb8b1ccab8929817603 (127.27.109.1:44095), 2f090ac440334007add53d66f47bd85d (127.27.109.3:41671)
I20260812 11:19:46.973482 23861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dac5497a4e4b44bf67491c912b7c8b" candidate_uuid: "5384228089e74241ba690c1190656dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f090ac440334007add53d66f47bd85d"
I20260812 11:19:46.973572 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dac5497a4e4b44bf67491c912b7c8b" candidate_uuid: "5384228089e74241ba690c1190656dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea5f000e80a4eb8b1ccab8929817603"
I20260812 11:19:46.973634 23861 raft_consensus.cc:3037] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:46.973721 23394 raft_consensus.cc:3037] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:46.975262 23861 raft_consensus.cc:2454] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5384228089e74241ba690c1190656dd7 in term 1.
I20260812 11:19:46.975262 23394 raft_consensus.cc:2454] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5384228089e74241ba690c1190656dd7 in term 1.
I20260812 11:19:46.975790 23516 leader_election.cc:302] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5384228089e74241ba690c1190656dd7, 8ea5f000e80a4eb8b1ccab8929817603; no voters:
I20260812 11:19:46.976164 23987 raft_consensus.cc:2781] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:46.976475 23987 raft_consensus.cc:686] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 5384228089e74241ba690c1190656dd7, State: Running, Role: LEADER
I20260812 11:19:46.977051 23987 consensus_queue.cc:237] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } }
I20260812 11:19:46.981812 23123 catalog_manager.cc:5495] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5384228089e74241ba690c1190656dd7 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "5384228089e74241ba690c1190656dd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5384228089e74241ba690c1190656dd7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 } health_report { overall_health: UNKNOWN } } }
W20260812 11:19:46.991771 23717 tablet.cc:2369] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:47.010578 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:47.005654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.27.109.2:45103)
0812 11:19:47.005871 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:47.005913 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:47.005936 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:19:47.006055 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:47.006058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:47.006125 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:47.006249 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:47.006650 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.007266 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.008467 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.008710 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.008861 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:47.009275 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:47.009912 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:19:47.010083 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:47.010273 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 11:19:47.010361 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 11:19:47.019147 23394 raft_consensus.cc:1261] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 1 FOLLOWER]: Refusing update from remote peer 5384228089e74241ba690c1190656dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:47.019232 23861 raft_consensus.cc:1261] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 1 FOLLOWER]: Refusing update from remote peer 5384228089e74241ba690c1190656dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:47.020485 23987 consensus_queue.cc:1035] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea5f000e80a4eb8b1ccab8929817603" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:47.020879 23993 consensus_queue.cc:1035] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f090ac440334007add53d66f47bd85d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:47.032466 24001 mvcc.cc:205] Tried to move back new op lower bound from 7317641572419026944 to 7317641572262010880. Current Snapshot: MvccSnapshot[applied={T|T < 7317641572419026944}]
I20260812 11:19:47.034188 24002 mvcc.cc:205] Tried to move back new op lower bound from 7317641572419026944 to 7317641572262010880. Current Snapshot: MvccSnapshot[applied={T|T < 7317641572419026944}]
I20260812 11:19:47.045522 23122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:47.045797 23122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:47.050365 23122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0cb589e4d364816b1704c3945caccc8: Sending DeleteTablet for 3 replicas of tablet 69dac5497a4e4b44bf67491c912b7c8b
I20260812 11:19:47.052085 23361 tablet_service.cc:1514] Processing DeleteTablet for tablet 69dac5497a4e4b44bf67491c912b7c8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:47 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:50854
I20260812 11:19:47.052117 23604 tablet_service.cc:1514] Processing DeleteTablet for tablet 69dac5497a4e4b44bf67491c912b7c8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:47 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:33764
I20260812 11:19:47.052479 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 69dac5497a4e4b44bf67491c912b7c8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:47 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:50794
I20260812 11:19:47.052961 24014 tablet_replica.cc:330] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603: stopping tablet replica
I20260812 11:19:47.053459 24015 tablet_replica.cc:330] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: stopping tablet replica
I20260812 11:19:47.053727 24013 tablet_replica.cc:330] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: stopping tablet replica
I20260812 11:19:47.053941 24015 raft_consensus.cc:2229] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:47.054111 24013 raft_consensus.cc:2229] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:19:47.054608 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 23240
I20260812 11:19:47.054654 24014 raft_consensus.cc:2229] T 69dac5497a4e4b44bf67491c912b7c8b P 8ea5f000e80a4eb8b1ccab8929817603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:47.153990 24013 raft_consensus.cc:2258] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:47.158782 24013 ts_tablet_manager.cc:1905] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:47.159447 24015 raft_consensus.cc:2258] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:47.162125 24015 ts_tablet_manager.cc:1905] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:47.164088 24013 ts_tablet_manager.cc:1918] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:47.164355 24013 log.cc:1196] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/69dac5497a4e4b44bf67491c912b7c8b
I20260812 11:19:47.165319 24013 ts_tablet_manager.cc:1939] T 69dac5497a4e4b44bf67491c912b7c8b P 5384228089e74241ba690c1190656dd7: Deleting consensus metadata
I20260812 11:19:47.167683 24015 ts_tablet_manager.cc:1918] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:47.167682 23101 catalog_manager.cc:4841] TS 5384228089e74241ba690c1190656dd7 (127.27.109.2:45103): tablet 69dac5497a4e4b44bf67491c912b7c8b (table test-table [id=c44f5a561b9249ef9eb93847c1fa517c]) successfully deleted
I20260812 11:19:47.167894 24015 log.cc:1196] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/69dac5497a4e4b44bf67491c912b7c8b
I20260812 11:19:47.168370 24015 ts_tablet_manager.cc:1939] T 69dac5497a4e4b44bf67491c912b7c8b P 2f090ac440334007add53d66f47bd85d: Deleting consensus metadata
W20260812 11:19:47.168555 23097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:44095 (error 108)
I20260812 11:19:47.168946 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 23481
I20260812 11:19:47.173485 23100 catalog_manager.cc:4841] TS 2f090ac440334007add53d66f47bd85d (127.27.109.3:41671): tablet 69dac5497a4e4b44bf67491c912b7c8b (table test-table [id=c44f5a561b9249ef9eb93847c1fa517c]) successfully deleted
I20260812 11:19:47.176491 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:47.176053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:44095 (local address 127.0.0.1:50864)
0812 11:19:47.176261 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:47.176373 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260812 11:19:47.176800 23097 catalog_manager.cc:4584] TS 8ea5f000e80a4eb8b1ccab8929817603 (127.27.109.1:44095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69dac5497a4e4b44bf67491c912b7c8b: Network error: Client connection negotiation failed: client connection to 127.27.109.1:44095: connect: Connection refused (error 111)
I20260812 11:19:47.183287 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 23723
I20260812 11:19:47.219116 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 23071
2026-08-12T11:19:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3419 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T11:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:19:47Z Disabled control of system clock
I20260812 11:19:47.278220 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41465
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:19:47.435189 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:47.435863 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:47.435979 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:47.445058 24020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:47.445214 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:47.445245 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:47.445287 24020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:47.445326 24020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:47.460448 24020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:47.462666 24020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:47.465744 24020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:47.478967 24026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:47.478940 24027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:47.478998 24033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:47.479252 24020 server_base.cc:1034] running on GCE node
I20260812 11:19:47.480386 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:47.482158 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:47.483515 24020 hybrid_clock.cc:648] HybridClock initialized: now 1786533587483479 us; error 56 us; skew 500 ppm
I20260812 11:19:47.486697 24020 webserver.cc:460] Webserver started at http://127.27.109.62:42569/ using document root <none> and password file <none>
I20260812 11:19:47.487782 24020 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:47.487892 24020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:47.488196 24020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:47.491518 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "f4f3bb11bd8e4b22ad48977a13c554bb"
format_stamp: "Formatted at 2026-08-12 11:19:47 on dist-test-slave-7z32"
server_key: "721973f52642e1c98a98f503edcccbcf"
server_key_iv: "3a780f6777f2598079125e56be19a2fe"
server_key_version: "encryptionkey@0"
I20260812 11:19:47.493098 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "f4f3bb11bd8e4b22ad48977a13c554bb"
format_stamp: "Formatted at 2026-08-12 11:19:47 on dist-test-slave-7z32"
server_key: "721973f52642e1c98a98f503edcccbcf"
server_key_iv: "3a780f6777f2598079125e56be19a2fe"
server_key_version: "encryptionkey@0"
I20260812 11:19:47.499966 24020 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 11:19:47.505837 24045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:47.508590 24020 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:19:47.508858 24020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "f4f3bb11bd8e4b22ad48977a13c554bb"
format_stamp: "Formatted at 2026-08-12 11:19:47 on dist-test-slave-7z32"
server_key: "721973f52642e1c98a98f503edcccbcf"
server_key_iv: "3a780f6777f2598079125e56be19a2fe"
server_key_version: "encryptionkey@0"
I20260812 11:19:47.509057 24020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:47.528506 24020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:47.529673 24020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:47.529917 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:47.541231 24020 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41465
I20260812 11:19:47.541240 24132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41465 every 8 connection(s)
I20260812 11:19:47.542788 24020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:47.547317 24134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:47.552253 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 24020
I20260812 11:19:47.552434 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:19:47.552790 28084 external_mini_cluster.cc:1428] Setting key 583359df0c68cbe3a0b2df29c7e6e1e5
I20260812 11:19:47.555351 24134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb: Bootstrap starting.
I20260812 11:19:47.558254 24134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:47.559351 24134 log.cc:824] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:47.561962 24134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb: No bootstrap required, opened a new log
I20260812 11:19:47.567327 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:47.555272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.27.109.62:41465)
0812 11:19:47.556341 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:47.556366 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:47.556422 (+ 56us) server_negotiation.cc:407] Connection header received
0812 11:19:47.557393 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:47.557419 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:47.558062 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:47.558636 (+ 574us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:47.559365 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.560717 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.562087 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.562806 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.563125 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:47.564416 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:47.564424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:47.564447 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:47.565144 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:47.565238 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:47.565270 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:47.565441 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:47.565626 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 11:19:47.566049 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":263,"threads_started":1}
I20260812 11:19:47.567715 24134 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41465 } }
I20260812 11:19:47.568166 24134 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:47.568223 24134 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f3bb11bd8e4b22ad48977a13c554bb, State: Initialized, Role: FOLLOWER
I20260812 11:19:47.568732 24134 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41465 } }
I20260812 11:19:47.568958 24134 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:47.569051 24134 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:47.569165 24134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:47.570363 24134 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41465 } }
I20260812 11:19:47.570803 24134 leader_election.cc:302] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f3bb11bd8e4b22ad48977a13c554bb; no voters:
I20260812 11:19:47.571326 24134 leader_election.cc:288] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:19:47.571535 24141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:47.571920 24141 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [term 1 LEADER]: Becoming Leader. State: Replica: f4f3bb11bd8e4b22ad48977a13c554bb, State: Running, Role: LEADER
I20260812 11:19:47.572422 24141 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41465 } }
I20260812 11:19:47.572892 24134 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:47.574992 24146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41465 } } }
I20260812 11:19:47.575182 24146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:47.574726 24147 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f3bb11bd8e4b22ad48977a13c554bb. Latest consensus state: current_term: 1 leader_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3bb11bd8e4b22ad48977a13c554bb" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41465 } } }
I20260812 11:19:47.575498 24147 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb [sys.catalog]: This master's current role is: LEADER
W20260812 11:19:47.578994 24168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:19:47.579131 24168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:19:47.579239 24165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:19:47.580147 24165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:19:47.588864 24165 catalog_manager.cc:1347] Generated new cluster ID: 416b279cb8d04b189a53c2f073393ef2
I20260812 11:19:47.589031 24165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:19:47.604823 24165 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:19:47.606629 24165 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:19:47.627009 24165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb: Generated new TSK 0
I20260812 11:19:47.627982 24165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:19:47.641620 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41465
--builtin_ntp_servers=127.27.109.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:19:47.830453 24173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:47.831058 24173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:47.831116 24173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:47.836165 24173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:47.836270 24173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:47.836339 24173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:47.842293 24173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:47.843948 24173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:47.846076 24173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:19:47.860164 24173 server_base.cc:1034] running on GCE node
W20260812 11:19:47.860062 24188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:47.860062 24185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:47.860069 24184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:47.861279 24173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:47.862291 24173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:47.863544 24173 hybrid_clock.cc:648] HybridClock initialized: now 1786533587863498 us; error 86 us; skew 500 ppm
I20260812 11:19:47.866019 24173 webserver.cc:460] Webserver started at http://127.27.109.1:43521/ using document root <none> and password file <none>
I20260812 11:19:47.866885 24173 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:47.867017 24173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:47.867326 24173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:47.870594 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "c5cd52d5c6b74913988bd04f441eb5b0"
format_stamp: "Formatted at 2026-08-12 11:19:47 on dist-test-slave-7z32"
server_key: "d34dc02522373a9e9319619931a0c63e"
server_key_iv: "f00abef897b9bd670f8eb672c52c3907"
server_key_version: "encryptionkey@0"
I20260812 11:19:47.871440 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "c5cd52d5c6b74913988bd04f441eb5b0"
format_stamp: "Formatted at 2026-08-12 11:19:47 on dist-test-slave-7z32"
server_key: "d34dc02522373a9e9319619931a0c63e"
server_key_iv: "f00abef897b9bd670f8eb672c52c3907"
server_key_version: "encryptionkey@0"
I20260812 11:19:47.876518 24173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:19:47.880169 24198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:47.882012 24173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:19:47.882197 24173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "c5cd52d5c6b74913988bd04f441eb5b0"
format_stamp: "Formatted at 2026-08-12 11:19:47 on dist-test-slave-7z32"
server_key: "d34dc02522373a9e9319619931a0c63e"
server_key_iv: "f00abef897b9bd670f8eb672c52c3907"
server_key_version: "encryptionkey@0"
I20260812 11:19:47.882326 24173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:47.900403 24173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:47.901211 24173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:47.901409 24173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:47.902038 24173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:47.903270 24173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:47.903329 24173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:47.903373 24173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:47.903420 24173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:47.915990 24173 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34955
I20260812 11:19:47.916019 24378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34955 every 8 connection(s)
I20260812 11:19:47.917128 24173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:47.926405 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 24173
I20260812 11:19:47.926644 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:19:47.927111 28084 external_mini_cluster.cc:1428] Setting key f967ea0f081d10b4b9334bb31b8aec14
I20260812 11:19:47.929544 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41465
--builtin_ntp_servers=127.27.109.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:19:47.933666 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:47.921105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:56837 (local address 127.27.109.62:41465)
0812 11:19:47.921536 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:47.921542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:47.923890 (+ 2348us) server_negotiation.cc:407] Connection header received
0812 11:19:47.926232 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:47.926236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:47.926340 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:47.926443 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:47.928199 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.928883 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.930599 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.930973 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.931134 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:47.932897 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:47.932901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:47.932904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:47.932930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:47.932957 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:47.932966 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:47.933071 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:47.933412 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 11:19:47.933526 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260812 11:19:47.935734 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:47.921805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41465 (local address 127.27.109.1:56837)
0812 11:19:47.923305 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:47.923566 (+ 261us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:47.925759 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:47.926905 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:47.926925 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:47.927078 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:47.927950 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:47.927971 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.929084 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.929093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:47.930418 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:47.930432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:47.931201 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:47.931206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:47.931454 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:47.932393 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:47.932396 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:47.932459 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:47.932634 (+ 175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:47.933064 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:47.933078 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:47.933087 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:19:47.933306 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 11:19:47.933784 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":190,"threads_started":1}
I20260812 11:19:47.937995 24379 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41465
I20260812 11:19:47.938498 24379 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:47.939680 24379 heartbeater.cc:511] Master 127.27.109.62:41465 requested a full tablet report, sending...
I20260812 11:19:47.942713 24069 ts_manager.cc:194] Registered new tserver with Master: c5cd52d5c6b74913988bd04f441eb5b0 (127.27.109.1:34955)
I20260812 11:19:47.945061 24069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:56837
W20260812 11:19:48.110111 24384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:48.110543 24384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:48.110591 24384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:48.116199 24384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:48.116307 24384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:48.116380 24384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:48.122325 24384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:48.124081 24384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:48.127017 24384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:48.143121 24392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:48.143342 24384 server_base.cc:1034] running on GCE node
W20260812 11:19:48.143112 24390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:48.143150 24389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:48.144331 24384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:48.145216 24384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:48.146652 24384 hybrid_clock.cc:648] HybridClock initialized: now 1786533588146600 us; error 80 us; skew 500 ppm
I20260812 11:19:48.149652 24384 webserver.cc:460] Webserver started at http://127.27.109.2:44563/ using document root <none> and password file <none>
I20260812 11:19:48.150396 24384 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:48.150477 24384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:48.150687 24384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:48.153646 24384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "18742bbfad74421f94190bb724ba60cb"
format_stamp: "Formatted at 2026-08-12 11:19:48 on dist-test-slave-7z32"
server_key: "cf2f148b8214dfc5a35677ce75726a82"
server_key_iv: "47c29fb9055630b7c6a9de676738ff56"
server_key_version: "encryptionkey@0"
I20260812 11:19:48.154445 24384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "18742bbfad74421f94190bb724ba60cb"
format_stamp: "Formatted at 2026-08-12 11:19:48 on dist-test-slave-7z32"
server_key: "cf2f148b8214dfc5a35677ce75726a82"
server_key_iv: "47c29fb9055630b7c6a9de676738ff56"
server_key_version: "encryptionkey@0"
I20260812 11:19:48.161113 24384 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 11:19:48.165069 24401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:48.166968 24384 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:19:48.167133 24384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "18742bbfad74421f94190bb724ba60cb"
format_stamp: "Formatted at 2026-08-12 11:19:48 on dist-test-slave-7z32"
server_key: "cf2f148b8214dfc5a35677ce75726a82"
server_key_iv: "47c29fb9055630b7c6a9de676738ff56"
server_key_version: "encryptionkey@0"
I20260812 11:19:48.167330 24384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:48.184635 24384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:48.185523 24384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:48.185786 24384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:48.186558 24384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:48.187846 24384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:48.187938 24384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:48.188006 24384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:48.188060 24384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:48.201148 24384 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44719
I20260812 11:19:48.201258 24595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44719 every 8 connection(s)
I20260812 11:19:48.202493 24384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:48.203143 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 24384
I20260812 11:19:48.203272 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:19:48.203573 28084 external_mini_cluster.cc:1428] Setting key e5053ea1a83ef5ef897c5de45f5840a8
I20260812 11:19:48.207110 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41465
--builtin_ntp_servers=127.27.109.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:19:48.221771 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.205954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:35511 (local address 127.27.109.62:41465)
0812 11:19:48.206169 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.206176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.208681 (+ 2505us) server_negotiation.cc:407] Connection header received
0812 11:19:48.212116 (+ 3435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.212120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.212229 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.212336 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:48.213990 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.214948 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.217593 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.217912 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.218082 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.220377 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:48.220383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:48.220386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:48.220412 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:48.220448 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:48.220462 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:48.220576 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.221176 (+ 600us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.221547 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 11:19:48.224864 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.206591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41465 (local address 127.27.109.2:35511)
0812 11:19:48.208319 (+ 1728us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.208430 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:48.211624 (+ 3194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:48.212655 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:48.212668 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:48.212815 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:48.213625 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.213646 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.215169 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.215174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.217301 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.217333 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.218192 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.218200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.218483 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.219674 (+ 1191us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:48.219677 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:48.219752 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:48.219901 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:48.220602 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:48.220791 (+ 189us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:48.220817 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:19:48.221147 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 11:19:48.222343 (+ 1196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":284,"threads_started":1}
I20260812 11:19:48.227399 24596 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41465
I20260812 11:19:48.227877 24596 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:48.228864 24596 heartbeater.cc:511] Master 127.27.109.62:41465 requested a full tablet report, sending...
I20260812 11:19:48.231050 24069 ts_manager.cc:194] Registered new tserver with Master: 18742bbfad74421f94190bb724ba60cb (127.27.109.2:44719)
I20260812 11:19:48.231989 24069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:35511
W20260812 11:19:48.367739 24600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:48.368093 24600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:48.368140 24600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:48.373351 24600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:48.373456 24600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:48.373522 24600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:48.379663 24600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:48.381223 24600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:48.383031 24600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:48.397080 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:48.397188 24605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:48.397257 24600 server_base.cc:1034] running on GCE node
W20260812 11:19:48.397166 24610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:48.398171 24600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:48.399088 24600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:48.400403 24600 hybrid_clock.cc:648] HybridClock initialized: now 1786533588400377 us; error 52 us; skew 500 ppm
I20260812 11:19:48.403252 24600 webserver.cc:460] Webserver started at http://127.27.109.3:46739/ using document root <none> and password file <none>
I20260812 11:19:48.404054 24600 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:48.404183 24600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:48.404664 24600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:48.407788 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "f9bef89122554179bd334e38b97a1874"
format_stamp: "Formatted at 2026-08-12 11:19:48 on dist-test-slave-7z32"
server_key: "78c7f7283177e1654b9247c4e09373f2"
server_key_iv: "a71f299407c7884929c74fad05ee8321"
server_key_version: "encryptionkey@0"
I20260812 11:19:48.408807 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "f9bef89122554179bd334e38b97a1874"
format_stamp: "Formatted at 2026-08-12 11:19:48 on dist-test-slave-7z32"
server_key: "78c7f7283177e1654b9247c4e09373f2"
server_key_iv: "a71f299407c7884929c74fad05ee8321"
server_key_version: "encryptionkey@0"
I20260812 11:19:48.416086 24600 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 11:19:48.419435 24619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:48.421191 24600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:19:48.421375 24600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "f9bef89122554179bd334e38b97a1874"
format_stamp: "Formatted at 2026-08-12 11:19:48 on dist-test-slave-7z32"
server_key: "78c7f7283177e1654b9247c4e09373f2"
server_key_iv: "a71f299407c7884929c74fad05ee8321"
server_key_version: "encryptionkey@0"
I20260812 11:19:48.421598 24600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:48.440270 24600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:48.441355 24600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:48.441669 24600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:48.442526 24600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:48.444298 24600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:48.444386 24600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:48.444916 24600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:48.444988 24600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:48.462765 24600 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:34319
I20260812 11:19:48.462913 24813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:34319 every 8 connection(s)
I20260812 11:19:48.464468 24600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:48.472901 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 24600
I20260812 11:19:48.473043 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:19:48.473491 28084 external_mini_cluster.cc:1428] Setting key 52eddd021b5dcb4f61b86deecab959d8
I20260812 11:19:48.480368 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.467280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:40911 (local address 127.27.109.62:41465)
0812 11:19:48.467492 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.467498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.469077 (+ 1579us) server_negotiation.cc:407] Connection header received
0812 11:19:48.471043 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.471047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.471121 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.471243 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:48.473357 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.474156 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.476641 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.477132 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.477280 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.479279 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:48.479283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:48.479286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:48.479305 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:48.479344 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:48.479411 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:48.479523 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.480135 (+ 612us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.480220 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 11:19:48.481953 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.467721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41465 (local address 127.27.109.3:40911)
0812 11:19:48.468812 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.468877 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:48.470626 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:48.471679 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:48.471695 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:48.471971 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:48.472938 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.472961 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.474464 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.474476 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.476410 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.476433 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.477382 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.477386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.477642 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.478708 (+ 1066us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:48.478710 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:48.478788 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:48.478993 (+ 205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:48.479586 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:48.479598 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:48.479807 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 11:19:48.480078 (+ 271us) client_negotiation.cc:240] Negotiation successful
0812 11:19:48.480578 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":184,"threads_started":1}
I20260812 11:19:48.484122 24814 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41465
I20260812 11:19:48.484623 24814 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:48.485534 24814 heartbeater.cc:511] Master 127.27.109.62:41465 requested a full tablet report, sending...
I20260812 11:19:48.488250 24069 ts_manager.cc:194] Registered new tserver with Master: f9bef89122554179bd334e38b97a1874 (127.27.109.3:34319)
I20260812 11:19:48.489264 24069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:40911
I20260812 11:19:48.489986 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:19:48.498364 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.493136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.27.109.62:41465)
0812 11:19:48.493294 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.493299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.493504 (+ 205us) server_negotiation.cc:407] Connection header received
0812 11:19:48.493601 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.493604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.493655 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.493755 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:48.494161 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.494768 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.495927 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.496192 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.496297 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.497770 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:48.497774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:48.497776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:48.497792 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:48.497820 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:48.497829 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:48.497923 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.498092 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.498168 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:19:48.503767 24068 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:19:48.507447 24068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:19:48.528225 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.521949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.27.109.2:44719)
0812 11:19:48.522305 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.522318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.522712 (+ 394us) server_negotiation.cc:407] Connection header received
0812 11:19:48.522815 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.522850 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.523210 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.523474 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:48.523950 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.525183 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.526605 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.527474 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.527615 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.527872 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.527929 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.528036 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
I20260812 11:19:48.528218 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.520965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44719 (local address 127.0.0.1:41020)
0812 11:19:48.522360 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.522422 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:48.522591 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:48.523675 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:48.523678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:48.523697 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:48.523822 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.523827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.525331 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.525334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.526466 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.526480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.527616 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.527620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.527733 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.527774 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 11:19:48.527891 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 11:19:48.528036 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"spinlock_wait_cycles":29952,"thread_start_us":129,"threads_started":1}
I20260812 11:19:48.528617 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.521502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:34319 (local address 127.0.0.1:35562)
0812 11:19:48.522737 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.522756 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:48.522909 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:48.524150 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:48.524154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:48.524175 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:48.524261 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.524267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.525519 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.525525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.526612 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.526631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.527833 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.527838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.528118 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.528168 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 11:19:48.528269 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 11:19:48.528408 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":105,"threads_started":1}
I20260812 11:19:48.530059 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.520700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34955 (local address 127.0.0.1:60000)
0812 11:19:48.521331 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.521396 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:48.521588 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:48.522502 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:48.522509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:48.522578 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:48.522804 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.522816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.524545 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.524549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.525984 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.525995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.529633 (+ 3638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.529637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.529740 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.529767 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:19:48.529839 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 11:19:48.529911 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"spinlock_wait_cycles":16640,"thread_start_us":133,"threads_started":1}
I20260812 11:19:48.530236 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.520968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.27.109.1:34955)
0812 11:19:48.521751 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.521760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.521798 (+ 38us) server_negotiation.cc:407] Connection header received
0812 11:19:48.521930 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.521936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.522233 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.522423 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:48.523013 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.524393 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.526168 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.529415 (+ 3247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.529586 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.529830 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.529903 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.530053 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":148,"threads_started":1}
I20260812 11:19:48.530941 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.522029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.27.109.3:34319)
0812 11:19:48.523597 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.523612 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.523645 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:19:48.523735 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.523741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.523956 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.524145 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:48.524446 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.525351 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.526795 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.527666 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.527825 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.528291 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.530560 (+ 2269us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.530722 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1433,"thread_start_us":187,"threads_started":1}
I20260812 11:19:48.533022 24481 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8fa313965e42648a9987660d1d6af4 (DEFAULT_TABLE table=test-table [id=168c39cb5546474b98ca15863883e1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:48.535041 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8fa313965e42648a9987660d1d6af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:48.535362 24270 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8fa313965e42648a9987660d1d6af4 (DEFAULT_TABLE table=test-table [id=168c39cb5546474b98ca15863883e1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:48.535257 24703 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8fa313965e42648a9987660d1d6af4 (DEFAULT_TABLE table=test-table [id=168c39cb5546474b98ca15863883e1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:48.536741 24703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8fa313965e42648a9987660d1d6af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:48.537142 24270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8fa313965e42648a9987660d1d6af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:48.549744 24841 tablet_bootstrap.cc:492] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Bootstrap starting.
I20260812 11:19:48.551400 24839 tablet_bootstrap.cc:492] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: Bootstrap starting.
I20260812 11:19:48.553088 24841 tablet_bootstrap.cc:654] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:48.554136 24840 tablet_bootstrap.cc:492] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Bootstrap starting.
I20260812 11:19:48.554390 24839 tablet_bootstrap.cc:654] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:48.554491 24841 log.cc:824] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:48.556860 24839 log.cc:824] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:48.558372 24841 tablet_bootstrap.cc:492] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: No bootstrap required, opened a new log
I20260812 11:19:48.558418 24840 tablet_bootstrap.cc:654] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:48.558713 24841 ts_tablet_manager.cc:1397] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 11:19:48.559530 24839 tablet_bootstrap.cc:492] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: No bootstrap required, opened a new log
I20260812 11:19:48.559844 24839 ts_tablet_manager.cc:1397] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 11:19:48.560300 24840 log.cc:824] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:48.563876 24840 tablet_bootstrap.cc:492] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: No bootstrap required, opened a new log
I20260812 11:19:48.564219 24840 ts_tablet_manager.cc:1397] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260812 11:19:48.564841 24839 raft_consensus.cc:348] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.564725 24841 raft_consensus.cc:348] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.565194 24839 raft_consensus.cc:374] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:48.565263 24839 raft_consensus.cc:729] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cd52d5c6b74913988bd04f441eb5b0, State: Initialized, Role: FOLLOWER
I20260812 11:19:48.565274 24841 raft_consensus.cc:374] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:48.565340 24841 raft_consensus.cc:729] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bef89122554179bd334e38b97a1874, State: Initialized, Role: FOLLOWER
I20260812 11:19:48.565837 24839 consensus_queue.cc:260] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.565953 24841 consensus_queue.cc:260] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.566704 24379 heartbeater.cc:503] Master 127.27.109.62:41465 was elected leader, sending a full tablet report...
I20260812 11:19:48.567513 24841 ts_tablet_manager.cc:1428] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Time spent starting tablet: real 0.009s user 0.010s sys 0.001s
I20260812 11:19:48.568114 24814 heartbeater.cc:503] Master 127.27.109.62:41465 was elected leader, sending a full tablet report...
I20260812 11:19:48.568903 24839 ts_tablet_manager.cc:1428] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20260812 11:19:48.572126 24840 raft_consensus.cc:348] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.572782 24840 raft_consensus.cc:374] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:48.572899 24840 raft_consensus.cc:729] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18742bbfad74421f94190bb724ba60cb, State: Initialized, Role: FOLLOWER
I20260812 11:19:48.573725 24840 consensus_queue.cc:260] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.574750 24596 heartbeater.cc:503] Master 127.27.109.62:41465 was elected leader, sending a full tablet report...
I20260812 11:19:48.575469 24840 ts_tablet_manager.cc:1428] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
W20260812 11:19:48.669261 24381 tablet.cc:2369] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:48.704617 24597 tablet.cc:2369] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:48.716121 24815 tablet.cc:2369] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:48.723094 24851 raft_consensus.cc:482] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:48.723413 24851 raft_consensus.cc:504] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.726015 24851 leader_election.cc:288] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18742bbfad74421f94190bb724ba60cb (127.27.109.2:44719), f9bef89122554179bd334e38b97a1874 (127.27.109.3:34319)
I20260812 11:19:48.733584 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.726593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:36727 (local address 127.27.109.2:44719)
0812 11:19:48.726782 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.726789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.727026 (+ 237us) server_negotiation.cc:407] Connection header received
0812 11:19:48.727366 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.727370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.727440 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.727544 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:48.728209 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.729237 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.731672 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.732423 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.732574 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.732639 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.733280 (+ 641us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.733430 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:19:48.733584 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.727078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:60485 (local address 127.27.109.3:34319)
0812 11:19:48.727227 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.727232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.727323 (+ 91us) server_negotiation.cc:407] Connection header received
0812 11:19:48.727386 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.727389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.727439 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.727544 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:48.728209 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.729237 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.731672 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.732378 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.732574 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.732639 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.733276 (+ 637us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.733391 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:19:48.733702 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.726356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44719 (local address 127.27.109.1:36727)
0812 11:19:48.726874 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.726897 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:48.727170 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:48.727621 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:48.727626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:48.727658 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:48.727973 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.727985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.729491 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.729496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.731408 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.731434 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.732653 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.732659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.732930 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.733085 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 11:19:48.733215 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 11:19:48.733395 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":353,"spinlock_wait_cycles":2432,"thread_start_us":263,"threads_started":1}
I20260812 11:19:48.733706 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.726643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:34319 (local address 127.27.109.1:60485)
0812 11:19:48.727118 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.727133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:48.727240 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:48.727676 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:48.727679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:48.727698 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:48.727985 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.727992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.729491 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.729496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.731408 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:48.731434 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.732653 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.732659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:48.732930 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.733057 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 11:19:48.733233 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 11:19:48.733395 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":312,"spinlock_wait_cycles":3968,"thread_start_us":200,"threads_started":1}
I20260812 11:19:48.735087 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8fa313965e42648a9987660d1d6af4" candidate_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18742bbfad74421f94190bb724ba60cb" is_pre_election: true
I20260812 11:19:48.735140 24742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8fa313965e42648a9987660d1d6af4" candidate_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bef89122554179bd334e38b97a1874" is_pre_election: true
I20260812 11:19:48.735608 24516 raft_consensus.cc:2454] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cd52d5c6b74913988bd04f441eb5b0 in term 0.
I20260812 11:19:48.735690 24742 raft_consensus.cc:2454] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cd52d5c6b74913988bd04f441eb5b0 in term 0.
I20260812 11:19:48.736545 24204 leader_election.cc:302] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18742bbfad74421f94190bb724ba60cb, c5cd52d5c6b74913988bd04f441eb5b0; no voters:
I20260812 11:19:48.737026 24851 raft_consensus.cc:2781] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:48.737149 24851 raft_consensus.cc:482] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:48.737455 24851 raft_consensus.cc:3037] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:48.739030 24851 raft_consensus.cc:504] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.739594 24851 leader_election.cc:288] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [CANDIDATE]: Term 1 election: Requested vote from peers 18742bbfad74421f94190bb724ba60cb (127.27.109.2:44719), f9bef89122554179bd334e38b97a1874 (127.27.109.3:34319)
I20260812 11:19:48.740372 24742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8fa313965e42648a9987660d1d6af4" candidate_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bef89122554179bd334e38b97a1874"
I20260812 11:19:48.740715 24742 raft_consensus.cc:3037] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:48.742441 24742 raft_consensus.cc:2454] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cd52d5c6b74913988bd04f441eb5b0 in term 1.
I20260812 11:19:48.743049 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8fa313965e42648a9987660d1d6af4" candidate_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18742bbfad74421f94190bb724ba60cb"
I20260812 11:19:48.743366 24203 leader_election.cc:302] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5cd52d5c6b74913988bd04f441eb5b0, f9bef89122554179bd334e38b97a1874; no voters:
I20260812 11:19:48.743314 24516 raft_consensus.cc:3037] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:48.743970 24851 raft_consensus.cc:2781] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:48.744418 24851 raft_consensus.cc:686] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 1 LEADER]: Becoming Leader. State: Replica: c5cd52d5c6b74913988bd04f441eb5b0, State: Running, Role: LEADER
I20260812 11:19:48.744881 24851 consensus_queue.cc:237] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } }
I20260812 11:19:48.746027 24516 raft_consensus.cc:2454] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cd52d5c6b74913988bd04f441eb5b0 in term 1.
I20260812 11:19:48.749203 24068 catalog_manager.cc:5495] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cd52d5c6b74913988bd04f441eb5b0 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cd52d5c6b74913988bd04f441eb5b0" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20260812 11:19:48.779729 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.774699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.27.109.1:34955)
0812 11:19:48.775010 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:48.775018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:48.775040 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:19:48.775418 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:48.775423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:48.775523 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:48.775695 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:48.776055 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.776643 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.777659 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:48.777909 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:48.778050 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:48.778471 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:48.779212 (+ 741us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:19:48.779372 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:48.779435 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 11:19:48.779518 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260812 11:19:48.790915 24516 raft_consensus.cc:1261] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 1 FOLLOWER]: Refusing update from remote peer c5cd52d5c6b74913988bd04f441eb5b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:48.790915 24742 raft_consensus.cc:1261] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 1 FOLLOWER]: Refusing update from remote peer c5cd52d5c6b74913988bd04f441eb5b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:48.792709 24858 consensus_queue.cc:1035] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bef89122554179bd334e38b97a1874" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:48.793259 24851 consensus_queue.cc:1035] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18742bbfad74421f94190bb724ba60cb" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:48.805925 24865 mvcc.cc:205] Tried to move back new op lower bound from 7317641579671969792 to 7317641579501932544. Current Snapshot: MvccSnapshot[applied={T|T < 7317641579671969792}]
I20260812 11:19:48.820245 24068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:48.820582 24068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:48.826719 24068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f3bb11bd8e4b22ad48977a13c554bb: Sending DeleteTablet for 3 replicas of tablet 5b8fa313965e42648a9987660d1d6af4
I20260812 11:19:48.829017 24481 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8fa313965e42648a9987660d1d6af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:48 UTC) from {username='slave'} at 127.0.0.1:41020
I20260812 11:19:48.829013 24270 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8fa313965e42648a9987660d1d6af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:48 UTC) from {username='slave'} at 127.0.0.1:60000
I20260812 11:19:48.829952 24703 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8fa313965e42648a9987660d1d6af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:48 UTC) from {username='slave'} at 127.0.0.1:35562
I20260812 11:19:48.829974 24875 tablet_replica.cc:330] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0: stopping tablet replica
I20260812 11:19:48.830533 24877 tablet_replica.cc:330] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: stopping tablet replica
I20260812 11:19:48.830562 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 24173
I20260812 11:19:48.830588 24875 raft_consensus.cc:2229] T 5b8fa313965e42648a9987660d1d6af4 P c5cd52d5c6b74913988bd04f441eb5b0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:19:48.831506 24876 tablet_replica.cc:330] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: stopping tablet replica
I20260812 11:19:48.832007 24877 raft_consensus.cc:2229] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:48.832442 24877 raft_consensus.cc:2258] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:48.832479 24876 raft_consensus.cc:2229] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:48.833084 24876 raft_consensus.cc:2258] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:48.839250 24876 ts_tablet_manager.cc:1905] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:48.839488 24877 ts_tablet_manager.cc:1905] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:48.843313 24877 ts_tablet_manager.cc:1918] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:48.843667 24877 log.cc:1196] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/5b8fa313965e42648a9987660d1d6af4
I20260812 11:19:48.844336 24877 ts_tablet_manager.cc:1939] T 5b8fa313965e42648a9987660d1d6af4 P f9bef89122554179bd334e38b97a1874: Deleting consensus metadata
I20260812 11:19:48.844400 24876 ts_tablet_manager.cc:1918] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:48.844882 24876 log.cc:1196] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/5b8fa313965e42648a9987660d1d6af4
I20260812 11:19:48.845567 24876 ts_tablet_manager.cc:1939] T 5b8fa313965e42648a9987660d1d6af4 P 18742bbfad74421f94190bb724ba60cb: Deleting consensus metadata
I20260812 11:19:48.847265 24049 catalog_manager.cc:4841] TS f9bef89122554179bd334e38b97a1874 (127.27.109.3:34319): tablet 5b8fa313965e42648a9987660d1d6af4 (table test-table [id=168c39cb5546474b98ca15863883e1dc]) successfully deleted
W20260812 11:19:48.849113 24051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:34955 (error 108)
I20260812 11:19:48.849289 24050 catalog_manager.cc:4841] TS 18742bbfad74421f94190bb724ba60cb (127.27.109.2:44719): tablet 5b8fa313965e42648a9987660d1d6af4 (table test-table [id=168c39cb5546474b98ca15863883e1dc]) successfully deleted
I20260812 11:19:48.850211 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 24384
I20260812 11:19:48.851774 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:48.850466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34955 (local address 127.0.0.1:54322)
0812 11:19:48.851474 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:48.851590 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":853}
W20260812 11:19:48.855118 24051 catalog_manager.cc:4584] TS c5cd52d5c6b74913988bd04f441eb5b0 (127.27.109.1:34955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b8fa313965e42648a9987660d1d6af4: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34955: connect: Connection refused (error 111)
I20260812 11:19:48.861958 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 24600
I20260812 11:19:48.896296 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 24020
2026-08-12T11:19:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1671 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:19:48 dist-test-slave-7z32 krb5kdc[24886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:19:48 dist-test-slave-7z32 krb5kdc[24886](info): set up 2 sockets
Aug 12 11:19:48 dist-test-slave-7z32 krb5kdc[24886](info): commencing operation
krb5kdc: starting...
W20260812 11:19:50.293725 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.359s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:19:50 dist-test-slave-7z32 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:19:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:19:50.476877 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33879
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:50.617622 24902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:50.617987 24902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:50.618032 24902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:50.623425 24902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:50.623545 24902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:50.623569 24902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:50.623589 24902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:50.623607 24902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:50.630752 24902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:50.632212 24902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:50.633446 24902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:50.644574 24908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:50.644543 24909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:50.644626 24902 server_base.cc:1034] running on GCE node
W20260812 11:19:50.644547 24912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:50.645968 24902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:50.648344 24902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:50.649940 24902 hybrid_clock.cc:648] HybridClock initialized: now 1786533590649856 us; error 99 us; skew 500 ppm
Aug 12 11:19:50 dist-test-slave-7z32 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533590, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:50.655640 24902 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:19:50.657378 24902 webserver.cc:460] Webserver started at http://127.27.109.62:35023/ using document root <none> and password file <none>
I20260812 11:19:50.658365 24902 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:50.658530 24902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:50.659242 24902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:50.665269 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "a1f84aad261144ce8f59f3da8bbe6d35"
format_stamp: "Formatted at 2026-08-12 11:19:50 on dist-test-slave-7z32"
server_key: "21a869a9a886fe7163597e4e89a80af6"
server_key_iv: "4200fb59333c79cd0e62a560698ee0bf"
server_key_version: "encryptionkey@0"
I20260812 11:19:50.667819 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "a1f84aad261144ce8f59f3da8bbe6d35"
format_stamp: "Formatted at 2026-08-12 11:19:50 on dist-test-slave-7z32"
server_key: "21a869a9a886fe7163597e4e89a80af6"
server_key_iv: "4200fb59333c79cd0e62a560698ee0bf"
server_key_version: "encryptionkey@0"
I20260812 11:19:50.674811 24902 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 11:19:50.679083 24925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:50.681337 24902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:19:50.681627 24902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "a1f84aad261144ce8f59f3da8bbe6d35"
format_stamp: "Formatted at 2026-08-12 11:19:50 on dist-test-slave-7z32"
server_key: "21a869a9a886fe7163597e4e89a80af6"
server_key_iv: "4200fb59333c79cd0e62a560698ee0bf"
server_key_version: "encryptionkey@0"
I20260812 11:19:50.681854 24902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:50.705121 24902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:50.709978 24902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:50.710346 24902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:50.721033 24902 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:33879
I20260812 11:19:50.721104 25011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:33879 every 8 connection(s)
I20260812 11:19:50.722543 24902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:50.726339 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:50.731150 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 24902
I20260812 11:19:50.731364 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:19:50.731740 28084 external_mini_cluster.cc:1428] Setting key 0b82438382acd45b49735464a38220dc
I20260812 11:19:50.734257 25012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35: Bootstrap starting.
I20260812 11:19:50.736902 25012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:50.737756 25012 log.cc:824] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:50.740520 25012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35: No bootstrap required, opened a new log
Aug 12 11:19:50 dist-test-slave-7z32 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:50.745060 25012 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33879 } }
I20260812 11:19:50.745344 25012 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:50.745381 25012 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f84aad261144ce8f59f3da8bbe6d35, State: Initialized, Role: FOLLOWER
I20260812 11:19:50.745947 25012 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33879 } }
I20260812 11:19:50.746089 25012 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:50.746198 25012 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:50.746357 25012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:50.747592 25012 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33879 } }
I20260812 11:19:50.748024 25012 leader_election.cc:302] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f84aad261144ce8f59f3da8bbe6d35; no voters:
I20260812 11:19:50.748512 25012 leader_election.cc:288] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:19:50.748605 25018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:50.748849 25018 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [term 1 LEADER]: Becoming Leader. State: Replica: a1f84aad261144ce8f59f3da8bbe6d35, State: Running, Role: LEADER
I20260812 11:19:50.749197 25018 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33879 } }
I20260812 11:19:50.750108 25012 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:50.751240 25019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33879 } } }
I20260812 11:19:50.751462 25019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:50.751811 25020 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f84aad261144ce8f59f3da8bbe6d35. Latest consensus state: current_term: 1 leader_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f84aad261144ce8f59f3da8bbe6d35" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33879 } } }
I20260812 11:19:50.751927 25020 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35 [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:50.752265 25034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:19:50.755206 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:50.733773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.27.109.62:33879)
0812 11:19:50.734409 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:50.734436 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:50.734727 (+ 291us) server_negotiation.cc:407] Connection header received
0812 11:19:50.735840 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:50.735864 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:50.736345 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:50.736808 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:50.738265 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:50.739726 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:50.740919 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:50.741604 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:50.741832 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:50.745880 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:50.745892 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:50.745925 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:50.745977 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:50.749048 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:50.750113 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:50.750122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:50.750131 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:50.750454 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:50.751208 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:50.751211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:50.751213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:50.752236 (+ 1023us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:50.753341 (+ 1105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:50.753649 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 11:19:50.754059 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":208,"threads_started":1}
I20260812 11:19:50.756506 25034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:19:50.764963 25034 catalog_manager.cc:1347] Generated new cluster ID: 62321b4647a44660aff834644061332c
I20260812 11:19:50.765090 25034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:19:50.781901 25034 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:19:50.783216 25034 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:19:50.787889 25034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35: Generated new TSK 0
I20260812 11:19:50.788798 25034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:19:50.844300 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33879
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:50.980626 25054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:50.980952 25054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:50.980999 25054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:50.985430 25054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:50.985525 25054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:50.985590 25054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:50.991673 25054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:50.992965 25054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:50.994040 25054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:51.005937 25060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:51.005930 25063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:51.006203 25054 server_base.cc:1034] running on GCE node
W20260812 11:19:51.006683 25061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:51.007375 25054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:51.008337 25054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:51.009637 25054 hybrid_clock.cc:648] HybridClock initialized: now 1786533591009607 us; error 57 us; skew 500 ppm
Aug 12 11:19:51 dist-test-slave-7z32 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533591, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:51.013374 25054 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:19:51.014534 25054 webserver.cc:460] Webserver started at http://127.27.109.1:41481/ using document root <none> and password file <none>
I20260812 11:19:51.015273 25054 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:51.015347 25054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:51.015606 25054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:51.017836 25054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "66df97c1abc1497e904b4107a41655bb"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "bbc0a6789f39ab13a15e369a9049fea8"
server_key_iv: "aeea258526740e406c5eb7f31bd0679f"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.018497 25054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "66df97c1abc1497e904b4107a41655bb"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "bbc0a6789f39ab13a15e369a9049fea8"
server_key_iv: "aeea258526740e406c5eb7f31bd0679f"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.023839 25054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:19:51.027968 25070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.030473 25054 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:19:51.030865 25054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "66df97c1abc1497e904b4107a41655bb"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "bbc0a6789f39ab13a15e369a9049fea8"
server_key_iv: "aeea258526740e406c5eb7f31bd0679f"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.031250 25054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:51.078516 25054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:51.082638 25054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:51.082970 25054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:51.083720 25054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:51.084841 25054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:51.084916 25054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.084992 25054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:51.085038 25054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.096817 25054 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37953
I20260812 11:19:51.097019 25247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37953 every 8 connection(s)
I20260812 11:19:51.097985 25054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:51.105540 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 25054
I20260812 11:19:51.105721 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:19:51.106117 28084 external_mini_cluster.cc:1428] Setting key 91ea8c52b51381398b741cb0ba63d482
Aug 12 11:19:51 dist-test-slave-7z32 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533591, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:51.118501 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.100944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:60583 (local address 127.27.109.62:33879)
0812 11:19:51.101211 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:51.101216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:51.102647 (+ 1431us) server_negotiation.cc:407] Connection header received
0812 11:19:51.104158 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:51.104161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:51.104226 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:51.104383 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:51.106740 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.107467 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.109042 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.109309 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.109429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.113719 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:51.113723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:51.113725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:51.113765 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:51.115657 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.116418 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.116422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.116424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.116523 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.116946 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.116950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.116952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.117212 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:51.117329 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:51.118054 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 11:19:51.118289 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 11:19:51.119758 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.101356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33879 (local address 127.27.109.1:60583)
0812 11:19:51.102379 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:51.102451 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:51.103824 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:51.104694 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:51.104705 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:51.105453 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:51.106461 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.106491 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.107666 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.107670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.108881 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.108897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.109503 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.109507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.109710 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.110595 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:51.110614 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:51.113409 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:51.115849 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:51.115860 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:51.115894 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:51.116286 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:51.116630 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:51.116633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:51.116636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:51.116835 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:51.117331 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:51.117344 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:51.117728 (+ 384us) client_negotiation.cc:769] Sending connection context
0812 11:19:51.117993 (+ 265us) client_negotiation.cc:240] Negotiation successful
0812 11:19:51.118495 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":167,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:19:51.121956 25249 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33879
I20260812 11:19:51.122524 25249 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260812 11:19:51.123379 25249 heartbeater.cc:511] Master 127.27.109.62:33879 requested a full tablet report, sending...
I20260812 11:19:51.126036 24951 ts_manager.cc:194] Registered new tserver with Master: 66df97c1abc1497e904b4107a41655bb (127.27.109.1:37953)
I20260812 11:19:51.127933 24951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:60583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:19:51.147856 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33879
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:51.284984 25255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:51.285482 25255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:51.285581 25255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:51.291441 25255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:51.291564 25255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:51.291673 25255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:51.298507 25255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:51.300215 25255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:51.302116 25255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:51.316803 25267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:51.316781 25262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:51.316756 25263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:51.317780 25255 server_base.cc:1034] running on GCE node
I20260812 11:19:51.318545 25255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:51.319628 25255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:51.320868 25255 hybrid_clock.cc:648] HybridClock initialized: now 1786533591320833 us; error 53 us; skew 500 ppm
Aug 12 11:19:51 dist-test-slave-7z32 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533591, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:51.325408 25255 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:19:51.327564 25255 webserver.cc:460] Webserver started at http://127.27.109.2:34921/ using document root <none> and password file <none>
I20260812 11:19:51.328516 25255 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:51.328624 25255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:51.328979 25255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:51.332305 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "2f0913c8815941998e7ede973d7beb53"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "93f7f2396e37464dcf7cd7d0b3a43fa4"
server_key_iv: "ae49feb376d7ae06dd59c662d638b0ba"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.333218 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "2f0913c8815941998e7ede973d7beb53"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "93f7f2396e37464dcf7cd7d0b3a43fa4"
server_key_iv: "ae49feb376d7ae06dd59c662d638b0ba"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.339437 25255 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 11:19:51.343848 25281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.346387 25255 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:19:51.346696 25255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "2f0913c8815941998e7ede973d7beb53"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "93f7f2396e37464dcf7cd7d0b3a43fa4"
server_key_iv: "ae49feb376d7ae06dd59c662d638b0ba"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.346997 25255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:51.375170 25255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:51.380667 25255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:51.380959 25255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:51.381781 25255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:51.383016 25255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:51.383131 25255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.383224 25255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:51.383281 25255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.398386 25255 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:33141
I20260812 11:19:51.398564 25474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:33141 every 8 connection(s)
I20260812 11:19:51.400260 25255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:51.401222 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 25255
I20260812 11:19:51.401381 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:19:51.401758 28084 external_mini_cluster.cc:1428] Setting key b9ddd813441d6c67e556fdfa998e158e
Aug 12 11:19:51 dist-test-slave-7z32 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533591, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:19:51.424842 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.403401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:43311 (local address 127.27.109.62:33879)
0812 11:19:51.403562 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:51.403567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:51.405761 (+ 2194us) server_negotiation.cc:407] Connection header received
0812 11:19:51.407164 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:51.407171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:51.407274 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:51.407431 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:51.409268 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.409938 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.412737 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.413032 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.413178 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.417963 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:51.417968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:51.417971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:51.418011 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:51.420563 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.421833 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.421841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.421852 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.422004 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.422793 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.422796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.422797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.423222 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:51.423330 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:51.424457 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0812 11:19:51.424642 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:19:51.426491 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.404112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33879 (local address 127.27.109.2:43311)
0812 11:19:51.405444 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:51.405542 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:51.406740 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:51.407653 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:51.407662 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:51.408156 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:51.409001 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.409028 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.410159 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.410166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.412454 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.412478 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.413310 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.413317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.413640 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.414570 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:51.414604 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:51.417511 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:51.420850 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:51.420863 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:51.420891 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:51.421502 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:51.422196 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:51.422201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:51.422232 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:51.422583 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:51.423384 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:51.423399 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:51.423986 (+ 587us) client_negotiation.cc:769] Sending connection context
0812 11:19:51.424324 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 11:19:51.424962 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":199,"threads_started":1}
I20260812 11:19:51.429034 25475 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33879
I20260812 11:19:51.429488 25475 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:51.430552 25475 heartbeater.cc:511] Master 127.27.109.62:33879 requested a full tablet report, sending...
I20260812 11:19:51.433197 24951 ts_manager.cc:194] Registered new tserver with Master: 2f0913c8815941998e7ede973d7beb53 (127.27.109.2:33141)
I20260812 11:19:51.434221 24951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:43311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:19:51.448673 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33879
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:51.619134 25483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:51.619875 25483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:51.620064 25483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:51.625423 25483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:51.625551 25483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:51.625627 25483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:51.633653 25483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:51.635598 25483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:51.637435 25483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:51.653481 25490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:51.653494 25494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:51.653522 25491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:51.654079 25483 server_base.cc:1034] running on GCE node
I20260812 11:19:51.654697 25483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:51.655642 25483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:51.657125 25483 hybrid_clock.cc:648] HybridClock initialized: now 1786533591657089 us; error 57 us; skew 500 ppm
Aug 12 11:19:51 dist-test-slave-7z32 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533591, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:51.661877 25483 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:19:51.663535 25483 webserver.cc:460] Webserver started at http://127.27.109.3:39161/ using document root <none> and password file <none>
I20260812 11:19:51.664638 25483 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:51.664809 25483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:51.665160 25483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:51.668543 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "8ef6579028074a01a684f5e4e6320045"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "9e8f57beeabfbbffe4a12ec1a42211a4"
server_key_iv: "7d9a2244a1a80525abd66d814a71dd73"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.669420 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "8ef6579028074a01a684f5e4e6320045"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "9e8f57beeabfbbffe4a12ec1a42211a4"
server_key_iv: "7d9a2244a1a80525abd66d814a71dd73"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.676900 25483 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 11:19:51.682490 25506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.684828 25483 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:19:51.685166 25483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "8ef6579028074a01a684f5e4e6320045"
format_stamp: "Formatted at 2026-08-12 11:19:51 on dist-test-slave-7z32"
server_key: "9e8f57beeabfbbffe4a12ec1a42211a4"
server_key_iv: "7d9a2244a1a80525abd66d814a71dd73"
server_key_version: "encryptionkey@0"
I20260812 11:19:51.685380 25483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:51.705976 25483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:51.710646 25483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:51.710994 25483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:51.712018 25483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:51.713344 25483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:51.713454 25483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.713577 25483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:51.713634 25483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:51.729852 25483 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:34209
I20260812 11:19:51.729874 25680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:34209 every 8 connection(s)
I20260812 11:19:51.731557 25483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:51.732668 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 25483
I20260812 11:19:51.732895 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:19:51.733309 28084 external_mini_cluster.cc:1428] Setting key b4a57d94c09591d5ce8b04eb8e083b8e
Aug 12 11:19:51 dist-test-slave-7z32 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533591, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:51.752394 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.735221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:60713 (local address 127.27.109.62:33879)
0812 11:19:51.735377 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:51.735382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:51.737284 (+ 1902us) server_negotiation.cc:407] Connection header received
0812 11:19:51.738567 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:51.738571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:51.738635 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:51.738726 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:51.740856 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.741529 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.743053 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.743328 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.743445 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.746952 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:51.746958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:51.746962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:51.747007 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:51.749303 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.750243 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.750248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.750249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.750391 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.750950 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.750960 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.750961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.751221 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:51.751364 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:51.752008 (+ 644us) server_negotiation.cc:299] Negotiation successful
0812 11:19:51.752191 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:19:51.753485 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.735849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33879 (local address 127.27.109.3:60713)
0812 11:19:51.737014 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:51.737134 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:51.738313 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:51.738980 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:51.738992 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:51.739667 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:51.740569 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.740597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.741746 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.741750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.742858 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.742870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.743550 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.743554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.743741 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.744528 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:51.744548 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:51.746519 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:51.749522 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:51.749530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:51.749550 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:51.750059 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:51.750541 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:51.750545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:51.750547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:51.750801 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:51.751386 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:51.751395 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:51.751712 (+ 317us) client_negotiation.cc:769] Sending connection context
0812 11:19:51.751940 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 11:19:51.752361 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":233,"threads_started":1}
I20260812 11:19:51.755826 25682 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33879
I20260812 11:19:51.756188 25682 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:51.756888 25682 heartbeater.cc:511] Master 127.27.109.62:33879 requested a full tablet report, sending...
I20260812 11:19:51.758589 24951 ts_manager.cc:194] Registered new tserver with Master: 8ef6579028074a01a684f5e4e6320045 (127.27.109.3:34209)
I20260812 11:19:51.759644 24951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:60713
I20260812 11:19:51.763342 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:19:51.778440 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.766573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.27.109.62:33879)
0812 11:19:51.766739 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:51.766743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:51.766895 (+ 152us) server_negotiation.cc:407] Connection header received
0812 11:19:51.767101 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:51.767104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:51.767156 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:51.767263 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:51.768336 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.768975 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.770064 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.770335 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.770447 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.773081 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:51.773085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:51.773087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:51.773128 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:51.775208 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.776537 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.776540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.776542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.776702 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:51.777340 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:51.777343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:51.777344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:51.777616 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:51.777759 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:51.778144 (+ 385us) server_negotiation.cc:299] Negotiation successful
0812 11:19:51.778283 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:19:51.783501 24951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:19:51.787928 24951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:19:51.810729 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.802727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37953 (local address 127.0.0.1:60744)
0812 11:19:51.803761 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:51.803833 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:51.804018 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:51.804730 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:51.804739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:51.804768 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:51.804943 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.804951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.806915 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.806920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.808575 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.808595 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.810055 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.810060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.810206 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.810264 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 11:19:51.810372 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 11:19:51.810540 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":116,"thread_start_us":234,"threads_started":1}
I20260812 11:19:51.811370 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.803869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:34209 (local address 127.0.0.1:39434)
0812 11:19:51.804520 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:51.804538 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:51.804651 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:51.805857 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:51.805860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:51.805880 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:51.805990 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.805997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.807552 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.807555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.809148 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.809162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.810876 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.810879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.810975 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.811005 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:19:51.811066 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 11:19:51.811164 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":186,"thread_start_us":139,"threads_started":1}
I20260812 11:19:51.811828 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.803117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.27.109.1:37953)
0812 11:19:51.804127 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:51.804136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:51.804168 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:19:51.804265 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:51.804272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:51.804526 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:51.805190 (+ 664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:51.805356 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.806705 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.808765 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.809859 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.811256 (+ 1397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.811460 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:51.811533 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 11:19:51.811646 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":166,"threads_started":1}
I20260812 11:19:51.812048 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.804256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.27.109.2:33141)
0812 11:19:51.804845 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:51.804860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:51.804891 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:19:51.804998 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:51.805005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:51.805253 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:51.805464 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:51.805767 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.806965 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.809320 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.810284 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.810406 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.810754 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:51.811724 (+ 970us) server_negotiation.cc:299] Negotiation successful
0812 11:19:51.811874 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":195,"threads_started":1}
I20260812 11:19:51.812251 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.802751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33141 (local address 127.0.0.1:40300)
0812 11:19:51.804321 (+ 1570us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:51.804340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:51.804472 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:51.805437 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:51.805440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:51.805461 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:51.805598 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.805608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.807137 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.807153 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.809156 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:51.809162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.810842 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.810869 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:51.811278 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.811301 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:19:51.811384 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 11:19:51.812092 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"mutex_wait_us":226,"thread_start_us":224,"threads_started":1}
I20260812 11:19:51.812265 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:51.804733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.27.109.3:34209)
0812 11:19:51.805255 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:51.805269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:51.805308 (+ 39us) server_negotiation.cc:407] Connection header received
0812 11:19:51.805395 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:51.805402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:51.805617 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:51.805851 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:51.806159 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.807384 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.809338 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:51.810518 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:51.811654 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:51.811902 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:51.811963 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 11:19:51.812085 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":201,"threads_started":1}
I20260812 11:19:51.815749 25585 tablet_service.cc:1467] Processing CreateTablet for tablet 1e24e2c4686344bb95a5c3399dd77eb7 (DEFAULT_TABLE table=test-table [id=f142737404d54e0c9782f738615f45da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:51.815743 25148 tablet_service.cc:1467] Processing CreateTablet for tablet 1e24e2c4686344bb95a5c3399dd77eb7 (DEFAULT_TABLE table=test-table [id=f142737404d54e0c9782f738615f45da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:51.815991 25365 tablet_service.cc:1467] Processing CreateTablet for tablet 1e24e2c4686344bb95a5c3399dd77eb7 (DEFAULT_TABLE table=test-table [id=f142737404d54e0c9782f738615f45da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:51.817183 25585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e24e2c4686344bb95a5c3399dd77eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:51.817180 25148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e24e2c4686344bb95a5c3399dd77eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:51.817512 25365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e24e2c4686344bb95a5c3399dd77eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:51.826502 25712 tablet_bootstrap.cc:492] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: Bootstrap starting.
I20260812 11:19:51.828255 25714 tablet_bootstrap.cc:492] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Bootstrap starting.
I20260812 11:19:51.828346 25713 tablet_bootstrap.cc:492] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Bootstrap starting.
I20260812 11:19:51.831128 25713 tablet_bootstrap.cc:654] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:51.831128 25712 tablet_bootstrap.cc:654] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:51.831923 25714 tablet_bootstrap.cc:654] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:51.832327 25712 log.cc:824] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:51.832492 25713 log.cc:824] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:51.833364 25714 log.cc:824] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:51.835268 25713 tablet_bootstrap.cc:492] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: No bootstrap required, opened a new log
I20260812 11:19:51.835268 25712 tablet_bootstrap.cc:492] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: No bootstrap required, opened a new log
I20260812 11:19:51.835525 25712 ts_tablet_manager.cc:1397] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260812 11:19:51.835525 25713 ts_tablet_manager.cc:1397] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260812 11:19:51.836827 25714 tablet_bootstrap.cc:492] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: No bootstrap required, opened a new log
I20260812 11:19:51.837173 25714 ts_tablet_manager.cc:1397] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 11:19:51.840691 25713 raft_consensus.cc:348] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:51.841123 25713 raft_consensus.cc:374] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:51.841198 25713 raft_consensus.cc:729] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef6579028074a01a684f5e4e6320045, State: Initialized, Role: FOLLOWER
I20260812 11:19:51.841174 25712 raft_consensus.cc:348] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:51.841419 25712 raft_consensus.cc:374] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:51.841503 25712 raft_consensus.cc:729] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66df97c1abc1497e904b4107a41655bb, State: Initialized, Role: FOLLOWER
I20260812 11:19:51.841305 25714 raft_consensus.cc:348] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:51.841810 25714 raft_consensus.cc:374] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:51.841892 25714 raft_consensus.cc:729] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0913c8815941998e7ede973d7beb53, State: Initialized, Role: FOLLOWER
I20260812 11:19:51.841879 25712 consensus_queue.cc:260] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:51.842089 25713 consensus_queue.cc:260] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:51.842550 25714 consensus_queue.cc:260] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:51.843197 25713 ts_tablet_manager.cc:1428] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260812 11:19:51.843197 25712 ts_tablet_manager.cc:1428] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260812 11:19:51.843323 25249 heartbeater.cc:503] Master 127.27.109.62:33879 was elected leader, sending a full tablet report...
I20260812 11:19:51.843617 25682 heartbeater.cc:503] Master 127.27.109.62:33879 was elected leader, sending a full tablet report...
I20260812 11:19:51.843854 25714 ts_tablet_manager.cc:1428] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 11:19:51.844110 25475 heartbeater.cc:503] Master 127.27.109.62:33879 was elected leader, sending a full tablet report...
W20260812 11:19:51.859771 25250 tablet.cc:2369] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:51.902422 25477 tablet.cc:2369] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:51.983172 25683 tablet.cc:2369] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:52.106971 25721 raft_consensus.cc:482] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:52.107203 25721 raft_consensus.cc:504] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:52.108685 25721 leader_election.cc:288] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f0913c8815941998e7ede973d7beb53 (127.27.109.2:33141), 66df97c1abc1497e904b4107a41655bb (127.27.109.1:37953)
I20260812 11:19:52.113600 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.109072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:36553 (local address 127.27.109.2:33141)
0812 11:19:52.109281 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:52.109288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:52.109312 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:19:52.109401 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:52.109407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:52.109491 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:52.109624 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:52.110127 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.111037 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.112411 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.112878 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.113046 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:52.113114 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:52.113350 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 11:19:52.113440 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:19:52.113600 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.108926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33141 (local address 127.27.109.3:36553)
0812 11:19:52.109150 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.109169 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:52.109282 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:52.109667 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:52.109671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:52.109696 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:52.109922 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:52.109933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.111175 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.111182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:52.112162 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:52.112173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.113042 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.113047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:52.113184 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:52.113207 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:19:52.113329 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 11:19:52.113424 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":34}
I20260812 11:19:52.114432 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.109193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37953 (local address 127.27.109.3:57493)
0812 11:19:52.109749 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.109768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:52.109869 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:52.110381 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:52.110385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:52.110406 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:52.110535 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:52.110572 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.111674 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.111679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:52.112914 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:52.112929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.113967 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.113971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:52.114066 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:52.114087 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:19:52.114207 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 11:19:52.114289 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":3,"thread_start_us":208,"threads_started":1}
I20260812 11:19:52.114557 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.109822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57493 (local address 127.27.109.1:37953)
0812 11:19:52.109982 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:52.109990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:52.110007 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:19:52.110135 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:52.110139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:52.110188 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:52.110306 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:52.110721 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.111504 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.113086 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.113580 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.113770 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:52.113860 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:52.114255 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 11:19:52.114409 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:19:52.114686 25396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e24e2c4686344bb95a5c3399dd77eb7" candidate_uuid: "8ef6579028074a01a684f5e4e6320045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0913c8815941998e7ede973d7beb53" is_pre_election: true
I20260812 11:19:52.115291 25396 raft_consensus.cc:2454] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef6579028074a01a684f5e4e6320045 in term 0.
I20260812 11:19:52.115410 25176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e24e2c4686344bb95a5c3399dd77eb7" candidate_uuid: "8ef6579028074a01a684f5e4e6320045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66df97c1abc1497e904b4107a41655bb" is_pre_election: true
I20260812 11:19:52.115720 25176 raft_consensus.cc:2454] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef6579028074a01a684f5e4e6320045 in term 0.
I20260812 11:19:52.116132 25513 leader_election.cc:302] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0913c8815941998e7ede973d7beb53, 8ef6579028074a01a684f5e4e6320045; no voters:
I20260812 11:19:52.116591 25721 raft_consensus.cc:2781] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:52.116734 25721 raft_consensus.cc:482] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:52.116801 25721 raft_consensus.cc:3037] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:52.118161 25721 raft_consensus.cc:504] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:52.118654 25721 leader_election.cc:288] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [CANDIDATE]: Term 1 election: Requested vote from peers 2f0913c8815941998e7ede973d7beb53 (127.27.109.2:33141), 66df97c1abc1497e904b4107a41655bb (127.27.109.1:37953)
I20260812 11:19:52.119330 25396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e24e2c4686344bb95a5c3399dd77eb7" candidate_uuid: "8ef6579028074a01a684f5e4e6320045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0913c8815941998e7ede973d7beb53"
I20260812 11:19:52.119330 25176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e24e2c4686344bb95a5c3399dd77eb7" candidate_uuid: "8ef6579028074a01a684f5e4e6320045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66df97c1abc1497e904b4107a41655bb"
I20260812 11:19:52.119537 25176 raft_consensus.cc:3037] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:52.119537 25396 raft_consensus.cc:3037] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:52.121333 25176 raft_consensus.cc:2454] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef6579028074a01a684f5e4e6320045 in term 1.
I20260812 11:19:52.121335 25396 raft_consensus.cc:2454] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef6579028074a01a684f5e4e6320045 in term 1.
I20260812 11:19:52.121937 25508 leader_election.cc:302] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66df97c1abc1497e904b4107a41655bb, 8ef6579028074a01a684f5e4e6320045; no voters:
I20260812 11:19:52.122299 25721 raft_consensus.cc:2781] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:52.122644 25721 raft_consensus.cc:686] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef6579028074a01a684f5e4e6320045, State: Running, Role: LEADER
I20260812 11:19:52.123199 25721 consensus_queue.cc:237] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } }
I20260812 11:19:52.127585 24949 catalog_manager.cc:5495] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ef6579028074a01a684f5e4e6320045 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "8ef6579028074a01a684f5e4e6320045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef6579028074a01a684f5e4e6320045" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34209 } health_report { overall_health: HEALTHY } } }
I20260812 11:19:52.195231 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.191004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.27.109.3:34209)
0812 11:19:52.191187 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:52.191194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:52.191214 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:19:52.191511 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:52.191514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:52.191578 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:52.191735 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:52.192036 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.192559 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.193507 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.193726 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.193860 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:52.194216 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:52.194682 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:19:52.194909 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:52.195014 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 11:19:52.195057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:19:52.204185 25396 raft_consensus.cc:1261] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 1 FOLLOWER]: Refusing update from remote peer 8ef6579028074a01a684f5e4e6320045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:52.204175 25176 raft_consensus.cc:1261] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb [term 1 FOLLOWER]: Refusing update from remote peer 8ef6579028074a01a684f5e4e6320045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:52.205376 25721 consensus_queue.cc:1035] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df97c1abc1497e904b4107a41655bb" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:52.205677 25726 consensus_queue.cc:1035] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0913c8815941998e7ede973d7beb53" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:52.216720 25739 mvcc.cc:205] Tried to move back new op lower bound from 7317641593654579200 to 7317641593340080128. Current Snapshot: MvccSnapshot[applied={T|T < 7317641593654579200}]
I20260812 11:19:52.227268 24949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:52.227520 24949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:52.231050 24949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f84aad261144ce8f59f3da8bbe6d35: Sending DeleteTablet for 3 replicas of tablet 1e24e2c4686344bb95a5c3399dd77eb7
I20260812 11:19:52.232244 25148 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e24e2c4686344bb95a5c3399dd77eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:52 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:60744
I20260812 11:19:52.232488 25365 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e24e2c4686344bb95a5c3399dd77eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:52 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40300
I20260812 11:19:52.233115 25585 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e24e2c4686344bb95a5c3399dd77eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:52 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:39434
I20260812 11:19:52.233309 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 25054
I20260812 11:19:52.233397 25753 tablet_replica.cc:330] T 1e24e2c4686344bb95a5c3399dd77eb7 P 66df97c1abc1497e904b4107a41655bb: stopping tablet replica
I20260812 11:19:52.234504 25756 tablet_replica.cc:330] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: stopping tablet replica
I20260812 11:19:52.235008 25756 raft_consensus.cc:2229] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:19:52.235520 25756 raft_consensus.cc:2258] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:52.242220 25756 ts_tablet_manager.cc:1905] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:52.246769 25755 tablet_replica.cc:330] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: stopping tablet replica
I20260812 11:19:52.247402 25756 ts_tablet_manager.cc:1918] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:52.247637 25756 log.cc:1196] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/1e24e2c4686344bb95a5c3399dd77eb7
I20260812 11:19:52.247761 25755 raft_consensus.cc:2229] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:52.248173 25756 ts_tablet_manager.cc:1939] T 1e24e2c4686344bb95a5c3399dd77eb7 P 8ef6579028074a01a684f5e4e6320045: Deleting consensus metadata
I20260812 11:19:52.248432 25755 raft_consensus.cc:2258] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:52.250026 24929 catalog_manager.cc:4841] TS 8ef6579028074a01a684f5e4e6320045 (127.27.109.3:34209): tablet 1e24e2c4686344bb95a5c3399dd77eb7 (table test-table [id=f142737404d54e0c9782f738615f45da]) successfully deleted
I20260812 11:19:52.251816 25755 ts_tablet_manager.cc:1905] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 11:19:52.255533 24926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:37953 (error 108)
I20260812 11:19:52.257015 25755 ts_tablet_manager.cc:1918] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:52.257066 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.256523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37953 (local address 127.0.0.1:60752)
0812 11:19:52.256679 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.256770 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 11:19:52.257238 25755 log.cc:1196] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/1e24e2c4686344bb95a5c3399dd77eb7
I20260812 11:19:52.257256 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 25255
I20260812 11:19:52.257648 25755 ts_tablet_manager.cc:1939] T 1e24e2c4686344bb95a5c3399dd77eb7 P 2f0913c8815941998e7ede973d7beb53: Deleting consensus metadata
W20260812 11:19:52.257640 24926 catalog_manager.cc:4584] TS 66df97c1abc1497e904b4107a41655bb (127.27.109.1:37953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e24e2c4686344bb95a5c3399dd77eb7: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37953: connect: Connection refused (error 111)
I20260812 11:19:52.278959 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 25483
I20260812 11:19:52.279232 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.278557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33141 (local address 127.0.0.1:40308)
0812 11:19:52.278881 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.278992 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.2:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 11:19:52.335336 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.334614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37953 (local address 127.0.0.1:60768)
0812 11:19:52.334978 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.335140 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 11:19:52.336697 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.336190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33141 (local address 127.0.0.1:40310)
0812 11:19:52.336461 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.336565 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.2:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260812 11:19:52.339160 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.338557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33141 (local address 127.0.0.1:40320)
0812 11:19:52.338925 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.339009 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.2:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260812 11:19:52.339223 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.338663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37953 (local address 127.0.0.1:60772)
0812 11:19:52.339019 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:52.339096 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260812 11:19:52.347873 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 24902
2026-08-12T11:19:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3456 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T11:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:19:52Z Disabled control of system clock
I20260812 11:19:52.408558 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39949
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:19:52.579577 25766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:52.580272 25766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:52.580394 25766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:52.586634 25766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:52.586843 25766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:52.586910 25766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:52.586956 25766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:52.586990 25766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:52.595964 25766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:52.598163 25766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:52.601063 25766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:52.614728 25778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:52.614738 25777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:52.615221 25781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:52.615190 25766 server_base.cc:1034] running on GCE node
I20260812 11:19:52.616129 25766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:52.617924 25766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:52.619200 25766 hybrid_clock.cc:648] HybridClock initialized: now 1786533592619160 us; error 67 us; skew 500 ppm
I20260812 11:19:52.627048 25766 webserver.cc:460] Webserver started at http://127.27.109.62:41259/ using document root <none> and password file <none>
I20260812 11:19:52.628019 25766 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:52.628159 25766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:52.628499 25766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:52.631995 25766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "c73d58898cf54c1f9e789227fb9ec8ff"
format_stamp: "Formatted at 2026-08-12 11:19:52 on dist-test-slave-7z32"
server_key: "6eb802bce6951e1821a0b6b743cfcee2"
server_key_iv: "5631ccf543cca0f1773c0b1061b56dc3"
server_key_version: "encryptionkey@0"
I20260812 11:19:52.632927 25766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "c73d58898cf54c1f9e789227fb9ec8ff"
format_stamp: "Formatted at 2026-08-12 11:19:52 on dist-test-slave-7z32"
server_key: "6eb802bce6951e1821a0b6b743cfcee2"
server_key_iv: "5631ccf543cca0f1773c0b1061b56dc3"
server_key_version: "encryptionkey@0"
I20260812 11:19:52.639223 25766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:19:52.643837 25789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:52.646136 25766 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:19:52.646462 25766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "c73d58898cf54c1f9e789227fb9ec8ff"
format_stamp: "Formatted at 2026-08-12 11:19:52 on dist-test-slave-7z32"
server_key: "6eb802bce6951e1821a0b6b743cfcee2"
server_key_iv: "5631ccf543cca0f1773c0b1061b56dc3"
server_key_version: "encryptionkey@0"
I20260812 11:19:52.646714 25766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:52.691982 25766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:52.693228 25766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:52.693526 25766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:52.705782 25766 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39949
I20260812 11:19:52.705943 25869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39949 every 8 connection(s)
I20260812 11:19:52.707592 25766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:52.712105 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:52.715276 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 25766
I20260812 11:19:52.715433 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:19:52.715869 28084 external_mini_cluster.cc:1428] Setting key 44922896ccbf34320b8a9c9d69e5e4c8
I20260812 11:19:52.720991 25871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff: Bootstrap starting.
I20260812 11:19:52.724282 25871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:52.725286 25871 log.cc:824] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:52.729810 25871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff: No bootstrap required, opened a new log
I20260812 11:19:52.730435 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:52.718374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.27.109.62:39949)
0812 11:19:52.719475 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:52.719499 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:52.719566 (+ 67us) server_negotiation.cc:407] Connection header received
0812 11:19:52.720446 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:52.720469 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:52.721023 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:52.721485 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:52.721986 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.723355 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.724798 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:52.725672 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:52.725913 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:52.726996 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:52.727008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:52.727038 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:52.728020 (+ 982us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:52.728170 (+ 150us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:52.728237 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:52.728550 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:52.728737 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 11:19:52.729273 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":267,"threads_started":1}
I20260812 11:19:52.735767 25871 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39949 } }
I20260812 11:19:52.736230 25871 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:52.736292 25871 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73d58898cf54c1f9e789227fb9ec8ff, State: Initialized, Role: FOLLOWER
I20260812 11:19:52.736936 25871 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39949 } }
I20260812 11:19:52.737129 25871 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:52.737182 25871 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:52.737289 25871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:52.738678 25871 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39949 } }
I20260812 11:19:52.739645 25871 leader_election.cc:302] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73d58898cf54c1f9e789227fb9ec8ff; no voters:
I20260812 11:19:52.740171 25871 leader_election.cc:288] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:19:52.740367 25878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:52.740763 25878 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [term 1 LEADER]: Becoming Leader. State: Replica: c73d58898cf54c1f9e789227fb9ec8ff, State: Running, Role: LEADER
I20260812 11:19:52.741214 25878 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39949 } }
I20260812 11:19:52.742635 25871 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:52.743835 25879 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73d58898cf54c1f9e789227fb9ec8ff. Latest consensus state: current_term: 1 leader_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39949 } } }
I20260812 11:19:52.743907 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73d58898cf54c1f9e789227fb9ec8ff" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39949 } } }
I20260812 11:19:52.744086 25879 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:52.744086 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:52.744869 25890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:19:52.749220 25890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:19:52.760334 25890 catalog_manager.cc:1347] Generated new cluster ID: 07d7a7f3e2964ad9bb17e6b6a250f59c
I20260812 11:19:52.760702 25890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:19:52.789978 25890 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:19:52.791988 25890 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:19:52.805694 25890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff: Generated new TSK 0
I20260812 11:19:52.807317 25890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:19:52.817425 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39949
--builtin_ntp_servers=127.27.109.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:19:52.974269 25907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:52.974682 25907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:52.974761 25907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:52.980032 25907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:52.980208 25907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:52.980327 25907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:52.987268 25907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:52.988790 25907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:52.990691 25907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:53.005510 25918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:53.005894 25915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:53.006060 25921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:53.006255 25907 server_base.cc:1034] running on GCE node
I20260812 11:19:53.006973 25907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:53.008011 25907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:53.011518 25907 hybrid_clock.cc:648] HybridClock initialized: now 1786533593011396 us; error 101 us; skew 500 ppm
I20260812 11:19:53.015242 25907 webserver.cc:460] Webserver started at http://127.27.109.1:42715/ using document root <none> and password file <none>
I20260812 11:19:53.016254 25907 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:53.016655 25907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:53.017129 25907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:53.021262 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "336a8207e91a40f39a7e5ba75aafdbc6"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "2799f561f654b9e4d08f8dc13e2391d0"
server_key_iv: "7346652eb735dcbefc20702264a369df"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.022485 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "336a8207e91a40f39a7e5ba75aafdbc6"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "2799f561f654b9e4d08f8dc13e2391d0"
server_key_iv: "7346652eb735dcbefc20702264a369df"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.033300 25907 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260812 11:19:53.038292 25929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.040738 25907 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:19:53.041036 25907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "336a8207e91a40f39a7e5ba75aafdbc6"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "2799f561f654b9e4d08f8dc13e2391d0"
server_key_iv: "7346652eb735dcbefc20702264a369df"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.041285 25907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:53.059480 25907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:53.060652 25907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:53.060940 25907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:53.062330 25907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:53.063874 25907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:53.063978 25907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.064064 25907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:53.064118 25907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.078975 25907 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34709
I20260812 11:19:53.079119 26111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34709 every 8 connection(s)
I20260812 11:19:53.080595 25907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:53.081398 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 25907
I20260812 11:19:53.081583 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:19:53.082422 28084 external_mini_cluster.cc:1428] Setting key 0db3df4bdc7e93cefaa5a7eb1409bbfa
I20260812 11:19:53.086185 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39949
--builtin_ntp_servers=127.27.109.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:19:53.098774 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.083719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:60749 (local address 127.27.109.62:39949)
0812 11:19:53.083945 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:53.083953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:53.085944 (+ 1991us) server_negotiation.cc:407] Connection header received
0812 11:19:53.088867 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:53.088874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:53.088974 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:53.089139 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:53.090892 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.091837 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.094092 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.094547 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.094748 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.097740 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:53.097748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:53.097753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:53.097783 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:53.097827 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:53.097845 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:53.098128 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:53.098417 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 11:19:53.098527 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 11:19:53.100662 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.084243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39949 (local address 127.27.109.1:60749)
0812 11:19:53.085635 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:53.085727 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:53.088315 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:53.089545 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:53.089556 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:53.089698 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:53.090593 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.090616 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.092161 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.092165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.093683 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.093707 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.095139 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.095155 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.095549 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.096887 (+ 1338us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:53.096890 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:53.096972 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:53.097374 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:53.098134 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:53.098151 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:53.098161 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:19:53.098364 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 11:19:53.098967 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":211,"threads_started":1}
I20260812 11:19:53.103286 26112 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39949
I20260812 11:19:53.103740 26112 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:53.104619 26112 heartbeater.cc:511] Master 127.27.109.62:39949 requested a full tablet report, sending...
I20260812 11:19:53.107705 25812 ts_manager.cc:194] Registered new tserver with Master: 336a8207e91a40f39a7e5ba75aafdbc6 (127.27.109.1:34709)
I20260812 11:19:53.109295 25812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:60749
W20260812 11:19:53.242511 26117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:53.242933 26117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:53.243023 26117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:53.247924 26117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:53.248023 26117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:53.248145 26117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:53.253919 26117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:53.255367 26117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:53.256916 26117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:53.269104 26130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:53.269104 26133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:53.269137 26128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:53.269697 26117 server_base.cc:1034] running on GCE node
I20260812 11:19:53.270318 26117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:53.271552 26117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:53.274986 26117 hybrid_clock.cc:648] HybridClock initialized: now 1786533593274848 us; error 144 us; skew 500 ppm
I20260812 11:19:53.278141 26117 webserver.cc:460] Webserver started at http://127.27.109.2:45125/ using document root <none> and password file <none>
I20260812 11:19:53.279328 26117 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:53.279479 26117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:53.279747 26117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:53.282395 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "2ac45c11e2f04921a24bcb55a1a2ceab"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "96332845cd4ca275ef61767e5eea382d"
server_key_iv: "ec0c3defea9dcfd2e483d4e9e2e5500c"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.283277 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "2ac45c11e2f04921a24bcb55a1a2ceab"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "96332845cd4ca275ef61767e5eea382d"
server_key_iv: "ec0c3defea9dcfd2e483d4e9e2e5500c"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.288574 26117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:19:53.292245 26141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.294287 26117 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:19:53.294466 26117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "2ac45c11e2f04921a24bcb55a1a2ceab"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "96332845cd4ca275ef61767e5eea382d"
server_key_iv: "ec0c3defea9dcfd2e483d4e9e2e5500c"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.294602 26117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:53.342156 26117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:53.343230 26117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:53.343441 26117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:53.344290 26117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:53.345731 26117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:53.345803 26117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.345860 26117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:53.345875 26117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.361089 26117 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44125
I20260812 11:19:53.361326 26319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44125 every 8 connection(s)
I20260812 11:19:53.362502 26117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:53.371207 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 26117
I20260812 11:19:53.371454 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:19:53.371906 28084 external_mini_cluster.cc:1428] Setting key bc19026fe766885fc54b5c5474c01207
I20260812 11:19:53.375263 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39949
--builtin_ntp_servers=127.27.109.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:19:53.378288 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.365928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:50279 (local address 127.27.109.62:39949)
0812 11:19:53.366148 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:53.366160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:53.367967 (+ 1807us) server_negotiation.cc:407] Connection header received
0812 11:19:53.370229 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:53.370233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:53.370395 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:53.370563 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:53.373199 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.373919 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.375800 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.376048 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.376198 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.377662 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:53.377666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:53.377668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:53.377686 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:53.377708 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:53.377717 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:53.377812 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:53.378015 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 11:19:53.378099 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 11:19:53.379472 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.365962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39949 (local address 127.27.109.2:50279)
0812 11:19:53.367435 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:53.367512 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:53.369728 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:53.371245 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:53.371266 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:53.371475 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:53.372652 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.372680 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.374196 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.374202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.375601 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.375627 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.376259 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.376264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.376479 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.377282 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:53.377284 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:53.377337 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:53.377426 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:53.377816 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:53.377829 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:53.377838 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:19:53.378002 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 11:19:53.378391 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":167,"threads_started":1}
I20260812 11:19:53.381798 26322 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39949
I20260812 11:19:53.382252 26322 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:53.383435 26322 heartbeater.cc:511] Master 127.27.109.62:39949 requested a full tablet report, sending...
I20260812 11:19:53.385747 25812 ts_manager.cc:194] Registered new tserver with Master: 2ac45c11e2f04921a24bcb55a1a2ceab (127.27.109.2:44125)
I20260812 11:19:53.386631 25812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:50279
W20260812 11:19:53.557152 26328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:53.557752 26328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:53.557853 26328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:53.563140 26328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:53.563328 26328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:53.563453 26328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:53.570245 26328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:53.572134 26328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:53.574180 26328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:53.591394 26339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:53.591560 26334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:53.591394 26336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:53.591713 26328 server_base.cc:1034] running on GCE node
I20260812 11:19:53.592535 26328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:53.593724 26328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:53.595108 26328 hybrid_clock.cc:648] HybridClock initialized: now 1786533593595073 us; error 44 us; skew 500 ppm
I20260812 11:19:53.598275 26328 webserver.cc:460] Webserver started at http://127.27.109.3:37767/ using document root <none> and password file <none>
I20260812 11:19:53.599242 26328 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:53.599368 26328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:53.599773 26328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:53.603183 26328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "d28284275d1a40e5bc3f0c86c57560aa"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "b9aa2f0c2f546b8eb8bc48653c1fb5b4"
server_key_iv: "547075597643d04c4b8d30507e1502c2"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.604001 26328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "d28284275d1a40e5bc3f0c86c57560aa"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "b9aa2f0c2f546b8eb8bc48653c1fb5b4"
server_key_iv: "547075597643d04c4b8d30507e1502c2"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.609804 26328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:19:53.614038 26350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.616441 26328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 11:19:53.616845 26328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "d28284275d1a40e5bc3f0c86c57560aa"
format_stamp: "Formatted at 2026-08-12 11:19:53 on dist-test-slave-7z32"
server_key: "b9aa2f0c2f546b8eb8bc48653c1fb5b4"
server_key_iv: "547075597643d04c4b8d30507e1502c2"
server_key_version: "encryptionkey@0"
I20260812 11:19:53.617102 26328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:53.671255 26328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:53.672473 26328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:53.672851 26328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:53.674033 26328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:53.675604 26328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:53.675704 26328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.675788 26328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:53.675838 26328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:53.691613 26328 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:41169
I20260812 11:19:53.691738 26533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:41169 every 8 connection(s)
I20260812 11:19:53.692884 26328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:53.701367 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 26328
I20260812 11:19:53.701567 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:19:53.701958 28084 external_mini_cluster.cc:1428] Setting key 93800526057e41a49296624f16359f9e
I20260812 11:19:53.710002 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.696602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57495 (local address 127.27.109.62:39949)
0812 11:19:53.697251 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:53.697259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:53.698219 (+ 960us) server_negotiation.cc:407] Connection header received
0812 11:19:53.700302 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:53.700309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:53.700665 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:53.700868 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:53.703992 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.704779 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.706660 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.707117 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.707261 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.709206 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:53.709212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:53.709215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:53.709241 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:53.709284 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:53.709298 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:53.709431 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:53.709689 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 11:19:53.709799 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260812 11:19:53.711983 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.696963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39949 (local address 127.27.109.3:57495)
0812 11:19:53.697963 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:53.698026 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:53.699687 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:53.701509 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:53.701523 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:53.702204 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:53.703721 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.703751 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.705157 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.705162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.706496 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.706512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.707399 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.707403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.707662 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.708667 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:53.708669 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:53.708725 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:53.708826 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:53.709487 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:53.709499 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:53.709505 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:19:53.709648 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 11:19:53.710598 (+ 950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":286,"threads_started":1}
I20260812 11:19:53.713901 26535 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39949
I20260812 11:19:53.714411 26535 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:53.715493 26535 heartbeater.cc:511] Master 127.27.109.62:39949 requested a full tablet report, sending...
I20260812 11:19:53.718000 25812 ts_manager.cc:194] Registered new tserver with Master: d28284275d1a40e5bc3f0c86c57560aa (127.27.109.3:41169)
I20260812 11:19:53.719270 25812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:57495
I20260812 11:19:53.720588 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:19:53.731420 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.724871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.27.109.62:39949)
0812 11:19:53.725143 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:53.725150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:53.725406 (+ 256us) server_negotiation.cc:407] Connection header received
0812 11:19:53.725631 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:53.725636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:53.725756 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:53.725885 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:53.726410 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.727184 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.728815 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.729107 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.729261 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.730617 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:53.730621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:53.730623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:53.730644 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:53.730672 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:53.730683 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:53.730916 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:53.731136 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 11:19:53.731205 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260812 11:19:53.738696 25810 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:19:53.744791 25810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:19:53.770787 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.763111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.27.109.1:34709)
0812 11:19:53.763865 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:53.763876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:53.764285 (+ 409us) server_negotiation.cc:407] Connection header received
0812 11:19:53.764372 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:53.764376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:53.764596 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:53.764799 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:53.765602 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.767125 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.768822 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.769505 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.769636 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.770145 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:53.770320 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 11:19:53.770522 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":149,"threads_started":1}
I20260812 11:19:53.772140 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.762723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34709 (local address 127.0.0.1:43256)
0812 11:19:53.763782 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:53.763860 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:53.764101 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:53.765021 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:53.765025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:53.765050 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:53.765399 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.765413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.767325 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.767330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.768639 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.768650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.769670 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.769674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.769816 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.769879 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 11:19:53.770121 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 11:19:53.771670 (+ 1549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":5,"spinlock_wait_cycles":1920,"thread_start_us":170,"threads_started":1}
I20260812 11:19:53.772576 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.762728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44125 (local address 127.0.0.1:53676)
0812 11:19:53.763919 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:53.763939 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:53.764102 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:53.764905 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:53.764911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:53.764937 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:53.765379 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.765390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.767893 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.767896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.769360 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.769373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.771541 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.771546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.771966 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.772001 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:19:53.772127 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 11:19:53.772250 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":1016,"spinlock_wait_cycles":2816,"thread_start_us":170,"threads_started":1}
I20260812 11:19:53.772655 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.763050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.27.109.2:44125)
0812 11:19:53.764132 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:53.764147 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:53.764200 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:19:53.764311 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:53.764316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:53.764597 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:53.764799 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:53.766173 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.767641 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.769549 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.770954 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.771146 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.771419 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:53.772287 (+ 868us) server_negotiation.cc:299] Negotiation successful
0812 11:19:53.772467 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":230,"threads_started":1}
I20260812 11:19:53.772759 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.764711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.27.109.3:41169)
0812 11:19:53.765417 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:53.765437 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:53.765473 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:19:53.765579 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:53.765584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:53.765856 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:53.766069 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:53.766900 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.768054 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.769821 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.770916 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.771034 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.771276 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:53.772461 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0812 11:19:53.772579 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":112,"threads_started":1}
I20260812 11:19:53.772243 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:53.764062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41169 (local address 127.0.0.1:48996)
0812 11:19:53.764603 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:53.764620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:53.764731 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:53.766262 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:53.766265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:53.766306 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:53.766439 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.766447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.768228 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.768232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.769657 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:53.769666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:53.771350 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:53.771354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:53.771522 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:53.771551 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:19:53.771635 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:19:53.771855 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":97,"thread_start_us":98,"threads_started":1}
I20260812 11:19:53.776681 26012 tablet_service.cc:1467] Processing CreateTablet for tablet 6874ea77d28d47e9953615437a05287e (DEFAULT_TABLE table=test-table [id=4402e2e720f244ff9681cc5de21f594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:53.777931 26012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6874ea77d28d47e9953615437a05287e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:53.777316 26215 tablet_service.cc:1467] Processing CreateTablet for tablet 6874ea77d28d47e9953615437a05287e (DEFAULT_TABLE table=test-table [id=4402e2e720f244ff9681cc5de21f594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:53.777316 26423 tablet_service.cc:1467] Processing CreateTablet for tablet 6874ea77d28d47e9953615437a05287e (DEFAULT_TABLE table=test-table [id=4402e2e720f244ff9681cc5de21f594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:53.778896 26215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6874ea77d28d47e9953615437a05287e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:53.778896 26423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6874ea77d28d47e9953615437a05287e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:53.787773 26564 tablet_bootstrap.cc:492] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6: Bootstrap starting.
I20260812 11:19:53.790112 26565 tablet_bootstrap.cc:492] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Bootstrap starting.
I20260812 11:19:53.790603 26564 tablet_bootstrap.cc:654] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:53.790791 26566 tablet_bootstrap.cc:492] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Bootstrap starting.
I20260812 11:19:53.792183 26564 log.cc:824] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:53.794469 26566 tablet_bootstrap.cc:654] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:53.794659 26565 tablet_bootstrap.cc:654] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:53.795662 26566 log.cc:824] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:53.796087 26565 log.cc:824] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:53.796272 26564 tablet_bootstrap.cc:492] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6: No bootstrap required, opened a new log
I20260812 11:19:53.796525 26564 ts_tablet_manager.cc:1397] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 11:19:53.798653 26565 tablet_bootstrap.cc:492] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: No bootstrap required, opened a new log
I20260812 11:19:53.798957 26565 ts_tablet_manager.cc:1397] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 11:19:53.799008 26566 tablet_bootstrap.cc:492] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: No bootstrap required, opened a new log
I20260812 11:19:53.799309 26566 ts_tablet_manager.cc:1397] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260812 11:19:53.802917 26564 raft_consensus.cc:348] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:53.803637 26564 raft_consensus.cc:374] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:53.803400 26565 raft_consensus.cc:348] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:53.803735 26564 raft_consensus.cc:729] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336a8207e91a40f39a7e5ba75aafdbc6, State: Initialized, Role: FOLLOWER
I20260812 11:19:53.803797 26565 raft_consensus.cc:374] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:53.803869 26565 raft_consensus.cc:729] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28284275d1a40e5bc3f0c86c57560aa, State: Initialized, Role: FOLLOWER
I20260812 11:19:53.804420 26565 consensus_queue.cc:260] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:53.804454 26564 consensus_queue.cc:260] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:53.804880 26566 raft_consensus.cc:348] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:53.805235 26566 raft_consensus.cc:374] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:53.805251 26535 heartbeater.cc:503] Master 127.27.109.62:39949 was elected leader, sending a full tablet report...
I20260812 11:19:53.805287 26566 raft_consensus.cc:729] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac45c11e2f04921a24bcb55a1a2ceab, State: Initialized, Role: FOLLOWER
I20260812 11:19:53.805735 26565 ts_tablet_manager.cc:1428] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 11:19:53.805951 26564 ts_tablet_manager.cc:1428] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260812 11:19:53.806476 26112 heartbeater.cc:503] Master 127.27.109.62:39949 was elected leader, sending a full tablet report...
I20260812 11:19:53.806437 26566 consensus_queue.cc:260] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:53.807628 26322 heartbeater.cc:503] Master 127.27.109.62:39949 was elected leader, sending a full tablet report...
I20260812 11:19:53.807941 26566 ts_tablet_manager.cc:1428] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260812 11:19:53.832820 26113 tablet.cc:2369] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:53.864390 26326 tablet.cc:2369] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:53.944142 26537 tablet.cc:2369] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:54.178088 26575 raft_consensus.cc:482] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:54.178421 26575 raft_consensus.cc:504] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:54.179805 26575 leader_election.cc:288] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 336a8207e91a40f39a7e5ba75aafdbc6 (127.27.109.1:34709), d28284275d1a40e5bc3f0c86c57560aa (127.27.109.3:41169)
I20260812 11:19:54.186758 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:54.180841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:47559 (local address 127.27.109.3:41169)
0812 11:19:54.180999 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:54.181016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:54.181149 (+ 133us) server_negotiation.cc:407] Connection header received
0812 11:19:54.181369 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:54.181376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:54.181458 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:54.181570 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:54.182353 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.183145 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.185244 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.185988 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.186194 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:54.186259 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:54.186518 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 11:19:54.186602 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 11:19:54.186775 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:54.180388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:41169 (local address 127.27.109.2:47559)
0812 11:19:54.180981 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:54.181004 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:54.181196 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:54.181774 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:54.181778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:54.181814 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:54.182044 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:54.182054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.183331 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.183336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:54.184878 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:54.184897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.186195 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.186199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:54.186340 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:54.186364 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:19:54.186482 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 11:19:54.186585 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":181,"threads_started":1}
I20260812 11:19:54.188093 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:54.180785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:53155 (local address 127.27.109.1:34709)
0812 11:19:54.180986 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:54.180996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:54.181704 (+ 708us) server_negotiation.cc:407] Connection header received
0812 11:19:54.181799 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:54.181803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:54.181861 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:54.182000 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:54.182693 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.184037 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.186026 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.186737 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.187015 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:54.187095 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:54.187796 (+ 701us) server_negotiation.cc:299] Negotiation successful
0812 11:19:54.187916 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:19:54.188117 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:54.180388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34709 (local address 127.27.109.2:53155)
0812 11:19:54.181297 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:54.181321 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:54.181501 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:54.182070 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:54.182073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:54.182101 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:54.182440 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:54.182449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.184286 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.184290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:54.185686 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:54.185706 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.187445 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.187450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:54.187582 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:54.187617 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:19:54.187761 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 11:19:54.187887 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":38,"thread_start_us":155,"threads_started":1}
I20260812 11:19:54.188511 26453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6874ea77d28d47e9953615437a05287e" candidate_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d28284275d1a40e5bc3f0c86c57560aa" is_pre_election: true
I20260812 11:19:54.189046 26453 raft_consensus.cc:2454] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac45c11e2f04921a24bcb55a1a2ceab in term 0.
I20260812 11:19:54.189141 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6874ea77d28d47e9953615437a05287e" candidate_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" is_pre_election: true
I20260812 11:19:54.189659 26040 raft_consensus.cc:2454] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac45c11e2f04921a24bcb55a1a2ceab in term 0.
I20260812 11:19:54.190166 26143 leader_election.cc:302] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac45c11e2f04921a24bcb55a1a2ceab, d28284275d1a40e5bc3f0c86c57560aa; no voters:
I20260812 11:19:54.190740 26575 raft_consensus.cc:2781] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:54.190915 26575 raft_consensus.cc:482] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:54.190972 26575 raft_consensus.cc:3037] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:54.192564 26575 raft_consensus.cc:504] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:54.193554 26575 leader_election.cc:288] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [CANDIDATE]: Term 1 election: Requested vote from peers 336a8207e91a40f39a7e5ba75aafdbc6 (127.27.109.1:34709), d28284275d1a40e5bc3f0c86c57560aa (127.27.109.3:41169)
I20260812 11:19:54.194370 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6874ea77d28d47e9953615437a05287e" candidate_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336a8207e91a40f39a7e5ba75aafdbc6"
I20260812 11:19:54.194895 26040 raft_consensus.cc:3037] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:54.194769 26453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6874ea77d28d47e9953615437a05287e" candidate_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d28284275d1a40e5bc3f0c86c57560aa"
I20260812 11:19:54.195253 26453 raft_consensus.cc:3037] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:54.196972 26453 raft_consensus.cc:2454] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac45c11e2f04921a24bcb55a1a2ceab in term 1.
I20260812 11:19:54.197366 26040 raft_consensus.cc:2454] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac45c11e2f04921a24bcb55a1a2ceab in term 1.
I20260812 11:19:54.198035 26143 leader_election.cc:302] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac45c11e2f04921a24bcb55a1a2ceab, d28284275d1a40e5bc3f0c86c57560aa; no voters:
I20260812 11:19:54.198764 26575 raft_consensus.cc:2781] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:54.199365 26575 raft_consensus.cc:686] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 1 LEADER]: Becoming Leader. State: Replica: 2ac45c11e2f04921a24bcb55a1a2ceab, State: Running, Role: LEADER
I20260812 11:19:54.200059 26575 consensus_queue.cc:237] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } }
I20260812 11:19:54.205783 25810 catalog_manager.cc:5495] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac45c11e2f04921a24bcb55a1a2ceab (127.27.109.2). New cstate: current_term: 1 leader_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac45c11e2f04921a24bcb55a1a2ceab" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 } health_report { overall_health: UNKNOWN } } }
I20260812 11:19:54.247305 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:54.239955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.27.109.2:44125)
0812 11:19:54.240229 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:54.240233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:54.240262 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:19:54.240474 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:54.240477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:54.240582 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:54.240752 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:54.241346 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.242032 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.243592 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:54.244241 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:54.244396 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:54.245260 (+ 864us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:54.245987 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:19:54.246182 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:54.246704 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 11:19:54.246825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260812 11:19:54.256600 26453 raft_consensus.cc:1261] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 1 FOLLOWER]: Refusing update from remote peer 2ac45c11e2f04921a24bcb55a1a2ceab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:54.256908 26040 raft_consensus.cc:1261] T 6874ea77d28d47e9953615437a05287e P 336a8207e91a40f39a7e5ba75aafdbc6 [term 1 FOLLOWER]: Refusing update from remote peer 2ac45c11e2f04921a24bcb55a1a2ceab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:54.257805 26575 consensus_queue.cc:1035] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28284275d1a40e5bc3f0c86c57560aa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 41169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:54.258204 26582 consensus_queue.cc:1035] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [LEADER]: Connected to new peer: Peer: permanent_uuid: "336a8207e91a40f39a7e5ba75aafdbc6" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:54.271091 26587 mvcc.cc:205] Tried to move back new op lower bound from 7317641602062467072 to 7317641601848029184. Current Snapshot: MvccSnapshot[applied={T|T < 7317641602062467072}]
I20260812 11:19:54.275547 26588 mvcc.cc:205] Tried to move back new op lower bound from 7317641602062467072 to 7317641601848029184. Current Snapshot: MvccSnapshot[applied={T|T < 7317641602062467072}]
I20260812 11:19:54.284729 25810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:54.285099 25810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:54.288664 25810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c73d58898cf54c1f9e789227fb9ec8ff: Sending DeleteTablet for 3 replicas of tablet 6874ea77d28d47e9953615437a05287e
I20260812 11:19:54.289894 26012 tablet_service.cc:1514] Processing DeleteTablet for tablet 6874ea77d28d47e9953615437a05287e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:54 UTC) from {username='slave'} at 127.0.0.1:43256
I20260812 11:19:54.290153 26215 tablet_service.cc:1514] Processing DeleteTablet for tablet 6874ea77d28d47e9953615437a05287e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:54 UTC) from {username='slave'} at 127.0.0.1:53676
I20260812 11:19:54.290560 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 25907
I20260812 11:19:54.322927 26423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6874ea77d28d47e9953615437a05287e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:54 UTC) from {username='slave'} at 127.0.0.1:48996
I20260812 11:19:54.324416 26603 tablet_replica.cc:330] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: stopping tablet replica
I20260812 11:19:54.325084 26603 raft_consensus.cc:2229] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:54.325711 26603 raft_consensus.cc:2258] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:54.330300 26602 tablet_replica.cc:330] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: stopping tablet replica
I20260812 11:19:54.330858 26602 raft_consensus.cc:2229] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:19:54.331380 26602 raft_consensus.cc:2258] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:54.331844 26603 ts_tablet_manager.cc:1905] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:54.333086 26602 ts_tablet_manager.cc:1905] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:54.336431 26603 ts_tablet_manager.cc:1918] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:54.336601 26603 log.cc:1196] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/6874ea77d28d47e9953615437a05287e
I20260812 11:19:54.337037 26603 ts_tablet_manager.cc:1939] T 6874ea77d28d47e9953615437a05287e P d28284275d1a40e5bc3f0c86c57560aa: Deleting consensus metadata
I20260812 11:19:54.338905 26602 ts_tablet_manager.cc:1918] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:54.339058 26602 log.cc:1196] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/6874ea77d28d47e9953615437a05287e
I20260812 11:19:54.339429 26602 ts_tablet_manager.cc:1939] T 6874ea77d28d47e9953615437a05287e P 2ac45c11e2f04921a24bcb55a1a2ceab: Deleting consensus metadata
I20260812 11:19:54.340585 25793 catalog_manager.cc:4841] TS d28284275d1a40e5bc3f0c86c57560aa (127.27.109.3:41169): tablet 6874ea77d28d47e9953615437a05287e (table test-table [id=4402e2e720f244ff9681cc5de21f594a]) successfully deleted
I20260812 11:19:54.340899 25793 catalog_manager.cc:4841] TS 2ac45c11e2f04921a24bcb55a1a2ceab (127.27.109.2:44125): tablet 6874ea77d28d47e9953615437a05287e (table test-table [id=4402e2e720f244ff9681cc5de21f594a]) successfully deleted
W20260812 11:19:54.342207 25796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:34709 (error 108)
I20260812 11:19:54.342690 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 26117
I20260812 11:19:54.377068 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:54.371662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34709 (local address 127.0.0.1:43272)
0812 11:19:54.376703 (+ 5041us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:54.376888 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4882,"thread_start_us":73,"threads_started":1}
W20260812 11:19:54.377482 25796 catalog_manager.cc:4584] TS 336a8207e91a40f39a7e5ba75aafdbc6 (127.27.109.1:34709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6874ea77d28d47e9953615437a05287e: Network error: Client connection negotiation failed: client connection to 127.27.109.1:34709: connect: Connection refused (error 111)
I20260812 11:19:54.384737 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 26328
I20260812 11:19:54.402565 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 25766
2026-08-12T11:19:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:19:54 dist-test-slave-7z32 krb5kdc[26609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:19:54 dist-test-slave-7z32 krb5kdc[26609](info): set up 2 sockets
Aug 12 11:19:54 dist-test-slave-7z32 krb5kdc[26609](info): commencing operation
krb5kdc: starting...
W20260812 11:19:56.017536 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.558s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:19:56 dist-test-slave-7z32 krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:19:56.242352 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35209
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:35209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:56.406769 26624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:56.407575 26624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:56.407692 26624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:56.417120 26624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:56.417366 26624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:56.417413 26624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:56.417465 26624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:56.417506 26624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:56.426318 26624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:35209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:56.428224 26624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:56.429857 26624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:56.444137 26632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:56.444137 26635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:56.444219 26624 server_base.cc:1034] running on GCE node
W20260812 11:19:56.444160 26637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:56.445495 26624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:56.447371 26624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:56.448691 26624 hybrid_clock.cc:648] HybridClock initialized: now 1786533596448614 us; error 95 us; skew 500 ppm
Aug 12 11:19:56 dist-test-slave-7z32 krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533596, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:56.452994 26624 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:19:56.454620 26624 webserver.cc:460] Webserver started at http://127.27.109.62:35861/ using document root <none> and password file <none>
I20260812 11:19:56.455489 26624 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:56.455580 26624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:56.455919 26624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:56.459115 26624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "45b006ba14b94763bd70d66e1d0cc7b7"
format_stamp: "Formatted at 2026-08-12 11:19:56 on dist-test-slave-7z32"
server_key: "0e0398e955615dfc09de39893c8ed36f"
server_key_iv: "4175e236604400feccc7fcff814b1732"
server_key_version: "encryptionkey@0"
I20260812 11:19:56.460230 26624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "45b006ba14b94763bd70d66e1d0cc7b7"
format_stamp: "Formatted at 2026-08-12 11:19:56 on dist-test-slave-7z32"
server_key: "0e0398e955615dfc09de39893c8ed36f"
server_key_iv: "4175e236604400feccc7fcff814b1732"
server_key_version: "encryptionkey@0"
I20260812 11:19:56.467389 26624 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 11:19:56.471716 26644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:56.474146 26624 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:19:56.474429 26624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "45b006ba14b94763bd70d66e1d0cc7b7"
format_stamp: "Formatted at 2026-08-12 11:19:56 on dist-test-slave-7z32"
server_key: "0e0398e955615dfc09de39893c8ed36f"
server_key_iv: "4175e236604400feccc7fcff814b1732"
server_key_version: "encryptionkey@0"
I20260812 11:19:56.474694 26624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:56.493997 26624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:56.499722 26624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:56.499975 26624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:56.511312 26624 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:35209
I20260812 11:19:56.511339 26734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:35209 every 8 connection(s)
I20260812 11:19:56.512921 26624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:56.515443 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 26624
I20260812 11:19:56.515599 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:19:56.516085 28084 external_mini_cluster.cc:1428] Setting key 2429b2c37f4b77d623f413a316a4f945
I20260812 11:19:56.517081 26736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:56.526008 26736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7: Bootstrap starting.
I20260812 11:19:56.529542 26736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7: Neither blocks nor log segments found. Creating new log.
Aug 12 11:19:56 dist-test-slave-7z32 krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:56.530645 26736 log.cc:824] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:56.533478 26736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7: No bootstrap required, opened a new log
I20260812 11:19:56.538057 26736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35209 } }
I20260812 11:19:56.538468 26736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:56.538578 26736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b006ba14b94763bd70d66e1d0cc7b7, State: Initialized, Role: FOLLOWER
I20260812 11:19:56.539276 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:56.518481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.27.109.62:35209)
0812 11:19:56.519275 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:56.519303 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:56.519370 (+ 67us) server_negotiation.cc:407] Connection header received
0812 11:19:56.520782 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:56.520817 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:56.521403 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:56.521951 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:56.523396 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:56.524659 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:56.525953 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:56.526753 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:56.527063 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:56.531455 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:56.531465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:56.531491 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:56.531547 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:56.534483 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:56.535372 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:56.535379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:56.535388 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:56.535575 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:56.536038 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:56.536042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:56.536044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:56.536707 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:56.537041 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:56.537460 (+ 419us) server_negotiation.cc:299] Negotiation successful
0812 11:19:56.537926 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":245,"threads_started":1}
I20260812 11:19:56.539405 26736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35209 } }
I20260812 11:19:56.539685 26736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:56.539824 26736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:56.539965 26736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:56.541334 26736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35209 } }
I20260812 11:19:56.541829 26736 leader_election.cc:302] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45b006ba14b94763bd70d66e1d0cc7b7; no voters:
I20260812 11:19:56.542251 26736 leader_election.cc:288] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:19:56.542479 26744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:56.543073 26744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [term 1 LEADER]: Becoming Leader. State: Replica: 45b006ba14b94763bd70d66e1d0cc7b7, State: Running, Role: LEADER
I20260812 11:19:56.543668 26744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35209 } }
I20260812 11:19:56.544389 26736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:56.546026 26745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35209 } } }
I20260812 11:19:56.546232 26745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:56.546639 26746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45b006ba14b94763bd70d66e1d0cc7b7. Latest consensus state: current_term: 1 leader_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b006ba14b94763bd70d66e1d0cc7b7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35209 } } }
I20260812 11:19:56.546741 26746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7 [sys.catalog]: This master's current role is: LEADER
W20260812 11:19:56.551467 26764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:19:56.551620 26764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:19:56.551716 26757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:19:56.552671 26757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:19:56.559962 26757 catalog_manager.cc:1347] Generated new cluster ID: 785075d1cfab42cd8672b488f98c24df
I20260812 11:19:56.560073 26757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:19:56.573028 26757 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:19:56.574602 26757 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:19:56.585348 26757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7: Generated new TSK 0
I20260812 11:19:56.586448 26757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260812 11:19:56.640307 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35209
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:56.832613 26778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:56.833284 26778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:56.833379 26778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:56.841468 26778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:56.841648 26778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:56.841742 26778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:56.850755 26778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:56.852654 26778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:56.854125 26778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:56.869110 26789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:56.869086 26786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:56.869166 26787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:56.869716 26778 server_base.cc:1034] running on GCE node
I20260812 11:19:56.870715 26778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:56.871841 26778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:56.873104 26778 hybrid_clock.cc:648] HybridClock initialized: now 1786533596873053 us; error 76 us; skew 500 ppm
Aug 12 11:19:56 dist-test-slave-7z32 krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533596, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:56.878142 26778 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260812 11:19:56.879748 26778 webserver.cc:460] Webserver started at http://127.27.109.1:37957/ using document root <none> and password file <none>
I20260812 11:19:56.880587 26778 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:56.880681 26778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:56.880982 26778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:56.883872 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "9f697a692e3e4467a9780b36216b8f98"
format_stamp: "Formatted at 2026-08-12 11:19:56 on dist-test-slave-7z32"
server_key: "7730a8dc3ef0e34fd68163a399006320"
server_key_iv: "4f58533338f96b03b344316ffccf24a3"
server_key_version: "encryptionkey@0"
I20260812 11:19:56.884577 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "9f697a692e3e4467a9780b36216b8f98"
format_stamp: "Formatted at 2026-08-12 11:19:56 on dist-test-slave-7z32"
server_key: "7730a8dc3ef0e34fd68163a399006320"
server_key_iv: "4f58533338f96b03b344316ffccf24a3"
server_key_version: "encryptionkey@0"
I20260812 11:19:56.889973 26778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:19:56.893875 26797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:56.896001 26778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 11:19:56.896169 26778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "9f697a692e3e4467a9780b36216b8f98"
format_stamp: "Formatted at 2026-08-12 11:19:56 on dist-test-slave-7z32"
server_key: "7730a8dc3ef0e34fd68163a399006320"
server_key_iv: "4f58533338f96b03b344316ffccf24a3"
server_key_version: "encryptionkey@0"
I20260812 11:19:56.896302 26778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:56.922242 26778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:56.927564 26778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:56.927862 26778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:56.928721 26778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:56.929883 26778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:56.929944 26778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:56.929996 26778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:56.930014 26778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:56.943305 26778 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:46573
I20260812 11:19:56.943558 26978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:46573 every 8 connection(s)
I20260812 11:19:56.944669 26778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:56.955256 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 26778
I20260812 11:19:56.955488 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:19:56.955956 28084 external_mini_cluster.cc:1428] Setting key 5d1a82f614dac965fcab4989b32a490a
Aug 12 11:19:56 dist-test-slave-7z32 krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533596, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:56.967190 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:56.947623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50537 (local address 127.27.109.62:35209)
0812 11:19:56.947885 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:56.947890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:56.949071 (+ 1181us) server_negotiation.cc:407] Connection header received
0812 11:19:56.950889 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:56.950894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:56.950977 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:56.951120 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:56.952850 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:56.953488 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:56.955168 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:56.955633 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:56.955777 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:56.960496 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:56.960506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:56.960513 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:56.960575 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:56.963102 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:56.964382 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:56.964387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:56.964389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:56.964814 (+ 425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:56.965483 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:56.965487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:56.965489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:56.965775 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:56.965927 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:56.966702 (+ 775us) server_negotiation.cc:299] Negotiation successful
0812 11:19:56.966971 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260812 11:19:56.968508 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:56.948069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35209 (local address 127.27.109.1:50537)
0812 11:19:56.948832 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:56.948900 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:56.950526 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:56.951383 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:56.951393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:56.951926 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:56.952616 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:56.952634 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:56.953675 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:56.953679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:56.954930 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:56.954950 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:56.955910 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:56.955915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:56.956216 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:56.957132 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:56.957165 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:56.960021 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:56.963365 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:56.963377 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:56.963405 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:56.963980 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:56.965026 (+ 1046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:56.965030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:56.965033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:56.965336 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:56.965930 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:56.965943 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:56.966366 (+ 423us) client_negotiation.cc:769] Sending connection context
0812 11:19:56.966646 (+ 280us) client_negotiation.cc:240] Negotiation successful
0812 11:19:56.967217 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":212,"threads_started":1}
I20260812 11:19:56.970916 26979 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35209
I20260812 11:19:56.971570 26979 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260812 11:19:56.972673 26979 heartbeater.cc:511] Master 127.27.109.62:35209 requested a full tablet report, sending...
I20260812 11:19:56.975479 26661 ts_manager.cc:194] Registered new tserver with Master: 9f697a692e3e4467a9780b36216b8f98 (127.27.109.1:46573)
I20260812 11:19:56.977500 26661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:50537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260812 11:19:57.009928 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35209
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:57.154187 26984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:57.154563 26984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:57.154614 26984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:57.159152 26984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:57.159256 26984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:57.159323 26984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:57.165508 26984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:57.167062 26984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:57.168329 26984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:57.181740 26991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:57.181682 26998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:57.181682 26993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:57.182542 26984 server_base.cc:1034] running on GCE node
I20260812 11:19:57.183450 26984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:57.184587 26984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:57.185894 26984 hybrid_clock.cc:648] HybridClock initialized: now 1786533597185857 us; error 59 us; skew 500 ppm
Aug 12 11:19:57 dist-test-slave-7z32 krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:57.190915 26984 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260812 11:19:57.192629 26984 webserver.cc:460] Webserver started at http://127.27.109.2:44947/ using document root <none> and password file <none>
I20260812 11:19:57.193516 26984 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:57.193650 26984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:57.193985 26984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:57.197099 26984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "7303e4a6a4334176a2de3fb34bdf7d7e"
format_stamp: "Formatted at 2026-08-12 11:19:57 on dist-test-slave-7z32"
server_key: "546ab083be1817ff5b0b771bc2692c53"
server_key_iv: "5e4e38fd50395ccca5b157a351467b26"
server_key_version: "encryptionkey@0"
I20260812 11:19:57.197988 26984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "7303e4a6a4334176a2de3fb34bdf7d7e"
format_stamp: "Formatted at 2026-08-12 11:19:57 on dist-test-slave-7z32"
server_key: "546ab083be1817ff5b0b771bc2692c53"
server_key_iv: "5e4e38fd50395ccca5b157a351467b26"
server_key_version: "encryptionkey@0"
I20260812 11:19:57.205428 26984 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 11:19:57.209867 27009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:57.212352 26984 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:19:57.212656 26984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "7303e4a6a4334176a2de3fb34bdf7d7e"
format_stamp: "Formatted at 2026-08-12 11:19:57 on dist-test-slave-7z32"
server_key: "546ab083be1817ff5b0b771bc2692c53"
server_key_iv: "5e4e38fd50395ccca5b157a351467b26"
server_key_version: "encryptionkey@0"
I20260812 11:19:57.212868 26984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:57.247434 26984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:57.252455 26984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:57.252828 26984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:57.253891 26984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:57.255275 26984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:57.255368 26984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:57.255456 26984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:57.255512 26984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:57.270510 26984 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:33603
I20260812 11:19:57.270649 27203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:33603 every 8 connection(s)
I20260812 11:19:57.271972 26984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:57.273905 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 26984
I20260812 11:19:57.274128 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:19:57.274608 28084 external_mini_cluster.cc:1428] Setting key 7e409aa994323dd571215d31e8430679
Aug 12 11:19:57 dist-test-slave-7z32 krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260812 11:19:57.294602 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.274807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:43071 (local address 127.27.109.62:35209)
0812 11:19:57.275097 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.275105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.277151 (+ 2046us) server_negotiation.cc:407] Connection header received
0812 11:19:57.279126 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.279134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.279220 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.279349 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:57.282066 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.282705 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.284344 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.284613 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.284771 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.289133 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:57.289137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:57.289140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:57.289179 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:57.291306 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:57.292227 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:57.292233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:57.292235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:57.292379 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:57.292883 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:57.292886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:57.292888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:57.293221 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:57.293379 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.294130 (+ 751us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.294338 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 11:19:57.295665 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.275297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35209 (local address 127.27.109.2:43071)
0812 11:19:57.276829 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:57.276938 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:57.278562 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:57.279732 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:57.279750 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:57.280668 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:57.281747 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.281774 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.282974 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.282978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.284147 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.284161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.284854 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.284860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.285108 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.286001 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:57.286038 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:57.288809 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:57.291580 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:57.291590 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:57.291609 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:57.291995 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:57.292552 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:57.292555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:57.292556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:57.292743 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:57.293380 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:57.293387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:57.293738 (+ 351us) client_negotiation.cc:769] Sending connection context
0812 11:19:57.293977 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 11:19:57.294430 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":216,"threads_started":1}
I20260812 11:19:57.297286 27206 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35209
I20260812 11:19:57.297610 27206 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:57.298203 27206 heartbeater.cc:511] Master 127.27.109.62:35209 requested a full tablet report, sending...
I20260812 11:19:57.300040 26661 ts_manager.cc:194] Registered new tserver with Master: 7303e4a6a4334176a2de3fb34bdf7d7e (127.27.109.2:33603)
I20260812 11:19:57.300861 26661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:43071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260812 11:19:57.320469 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35209
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:19:57.483834 27216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:57.484277 27216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:57.484331 27216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:57.489347 27216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:57.489483 27216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:57.489558 27216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:57.496676 27216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:35209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:57.498934 27216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:57.501847 27216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:57.518131 27223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:57.518148 27221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:57.518152 27228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:57.518496 27216 server_base.cc:1034] running on GCE node
I20260812 11:19:57.519460 27216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:57.520434 27216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:57.523819 27216 hybrid_clock.cc:648] HybridClock initialized: now 1786533597523751 us; error 88 us; skew 500 ppm
Aug 12 11:19:57 dist-test-slave-7z32 krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:19:57.528304 27216 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260812 11:19:57.529618 27216 webserver.cc:460] Webserver started at http://127.27.109.3:34327/ using document root <none> and password file <none>
I20260812 11:19:57.530423 27216 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:57.530550 27216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:57.530874 27216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:57.533548 27216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "10e4b8e28ea84c5c808ec346f468fefb"
format_stamp: "Formatted at 2026-08-12 11:19:57 on dist-test-slave-7z32"
server_key: "3ed9cbc65fc0c9c350cb110767bd7fa2"
server_key_iv: "ce1a0015fb4ebd4c9ee9bab419e7bc54"
server_key_version: "encryptionkey@0"
I20260812 11:19:57.534353 27216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "10e4b8e28ea84c5c808ec346f468fefb"
format_stamp: "Formatted at 2026-08-12 11:19:57 on dist-test-slave-7z32"
server_key: "3ed9cbc65fc0c9c350cb110767bd7fa2"
server_key_iv: "ce1a0015fb4ebd4c9ee9bab419e7bc54"
server_key_version: "encryptionkey@0"
I20260812 11:19:57.540081 27216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 11:19:57.544032 27238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:57.546511 27216 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:19:57.546741 27216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "10e4b8e28ea84c5c808ec346f468fefb"
format_stamp: "Formatted at 2026-08-12 11:19:57 on dist-test-slave-7z32"
server_key: "3ed9cbc65fc0c9c350cb110767bd7fa2"
server_key_iv: "ce1a0015fb4ebd4c9ee9bab419e7bc54"
server_key_version: "encryptionkey@0"
I20260812 11:19:57.547060 27216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:57.562399 27216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:57.567513 27216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:57.567798 27216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:57.568894 27216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:57.570068 27216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:57.570132 27216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:57.570232 27216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:57.570274 27216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:57.584941 27216 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:39525
I20260812 11:19:57.585045 27423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:39525 every 8 connection(s)
I20260812 11:19:57.586221 27216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:57.595628 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 27216
I20260812 11:19:57.595837 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:19:57.596228 28084 external_mini_cluster.cc:1428] Setting key 14f3e1ec75eae3e97ae13b2d4d975588
Aug 12 11:19:57 dist-test-slave-7z32 krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:19:57.611512 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.589288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:59227 (local address 127.27.109.62:35209)
0812 11:19:57.589521 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.589528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.591447 (+ 1919us) server_negotiation.cc:407] Connection header received
0812 11:19:57.592642 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.592646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.592729 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.592860 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:57.595279 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.596267 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.598249 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.598543 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.598668 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.605178 (+ 6510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:57.605185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:57.605188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:57.605246 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:57.607699 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:57.608844 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:57.608858 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:57.608860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:57.609068 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:57.609748 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:57.609752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:57.609753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:57.610063 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:57.610177 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.611094 (+ 917us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.611322 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 11:19:57.612663 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.589886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35209 (local address 127.27.109.3:59227)
0812 11:19:57.591160 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:57.591232 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:57.592295 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:57.593127 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:57.593136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:57.593733 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:57.594733 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.594753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.596607 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.596612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.597958 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.597977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.599054 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.599060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.599342 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.601622 (+ 2280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:19:57.601649 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:57.604570 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:57.608046 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:57.608057 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:57.608080 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:57.608592 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:57.609239 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:19:57.609243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:19:57.609244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:19:57.609470 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:19:57.610219 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:57.610227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:57.610589 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 11:19:57.610812 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 11:19:57.611389 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":210,"threads_started":1}
I20260812 11:19:57.614655 27424 heartbeater.cc:347] Connected to a master server at 127.27.109.62:35209
I20260812 11:19:57.615119 27424 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:57.616067 27424 heartbeater.cc:511] Master 127.27.109.62:35209 requested a full tablet report, sending...
I20260812 11:19:57.617955 26661 ts_manager.cc:194] Registered new tserver with Master: 10e4b8e28ea84c5c808ec346f468fefb (127.27.109.3:39525)
I20260812 11:19:57.619047 26661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:59227
I20260812 11:19:57.628257 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:19:57.650517 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.631700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.27.109.62:35209)
0812 11:19:57.631886 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.631890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.632069 (+ 179us) server_negotiation.cc:407] Connection header received
0812 11:19:57.632131 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.632134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.632190 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.632357 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:57.633500 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.634128 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.635713 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.635991 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.636133 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.644732 (+ 8599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:57.644737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:57.644740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:19:57.644797 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:57.647251 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:57.648174 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:57.648178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:57.648180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:57.648294 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:19:57.648769 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:19:57.648774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:19:57.648776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:19:57.649392 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:57.649548 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.649959 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.650190 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:19:57.656527 26661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:19:57.661526 26661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:19:57.683858 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.675225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46573 (local address 127.0.0.1:43776)
0812 11:19:57.675751 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:57.675824 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:57.675982 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:57.678250 (+ 2268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:57.678276 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:57.678296 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:57.678647 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.678655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.680200 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.680204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.681913 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.681928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.683182 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.683186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.683339 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.683409 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 11:19:57.683522 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:19:57.683691 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":679,"thread_start_us":182,"threads_started":1}
I20260812 11:19:57.684899 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.675460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33603 (local address 127.0.0.1:41282)
0812 11:19:57.676672 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:57.676689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:57.676808 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:57.677980 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:57.677987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:57.678034 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:57.678332 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.678343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.680342 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.680346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.683013 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.683025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.684350 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.684355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.684460 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.684488 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:19:57.684551 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 11:19:57.684679 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":778,"spinlock_wait_cycles":6272,"thread_start_us":191,"threads_started":1}
I20260812 11:19:57.685348 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.676391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.27.109.1:46573)
0812 11:19:57.677271 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.677284 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.677329 (+ 45us) server_negotiation.cc:407] Connection header received
0812 11:19:57.677442 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.677448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.677941 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.678265 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:57.678868 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.680029 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.682098 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.683019 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.684779 (+ 1760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.684940 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.685067 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.685203 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":147,"threads_started":1}
I20260812 11:19:57.685392 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.676974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.27.109.3:39525)
0812 11:19:57.677357 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.677372 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.677401 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:19:57.677504 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.677510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.677766 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.677950 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:57.678546 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.679964 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.682926 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.684115 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.684291 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.684537 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.685003 (+ 466us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.685198 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
I20260812 11:19:57.685683 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.676327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39525 (local address 127.0.0.1:50308)
0812 11:19:57.676764 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:57.676785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:57.677174 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:57.677983 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:57.677999 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:57.678045 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:57.678291 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.678302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.680241 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.680251 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.682587 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.682623 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.684609 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.684612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.684693 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.684714 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:19:57.684787 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:19:57.684861 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":394,"spinlock_wait_cycles":2304,"thread_start_us":155,"threads_started":1}
I20260812 11:19:57.686028 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.676315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.27.109.2:33603)
0812 11:19:57.677509 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.677531 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.677553 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:19:57.677621 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.677625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.677766 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.678514 (+ 748us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:57.678699 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.680014 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.683237 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.684167 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.685361 (+ 1194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.685618 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.685686 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.685840 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":155,"threads_started":1}
I20260812 11:19:57.689841 27094 tablet_service.cc:1467] Processing CreateTablet for tablet 916afe0454c346fc8c316f6454a430eb (DEFAULT_TABLE table=test-table [id=964f1538dc4f4b3cbdcb8eec4060a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:57.690562 26884 tablet_service.cc:1467] Processing CreateTablet for tablet 916afe0454c346fc8c316f6454a430eb (DEFAULT_TABLE table=test-table [id=964f1538dc4f4b3cbdcb8eec4060a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:57.690635 27328 tablet_service.cc:1467] Processing CreateTablet for tablet 916afe0454c346fc8c316f6454a430eb (DEFAULT_TABLE table=test-table [id=964f1538dc4f4b3cbdcb8eec4060a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:57.691752 27094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916afe0454c346fc8c316f6454a430eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:57.692266 26884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916afe0454c346fc8c316f6454a430eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:57.692269 27328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916afe0454c346fc8c316f6454a430eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:57.699998 27450 tablet_bootstrap.cc:492] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Bootstrap starting.
I20260812 11:19:57.703416 27450 tablet_bootstrap.cc:654] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:57.704533 27450 log.cc:824] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:57.707194 27453 tablet_bootstrap.cc:492] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Bootstrap starting.
I20260812 11:19:57.707194 27452 tablet_bootstrap.cc:492] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: Bootstrap starting.
I20260812 11:19:57.708334 27450 tablet_bootstrap.cc:492] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: No bootstrap required, opened a new log
I20260812 11:19:57.708691 27450 ts_tablet_manager.cc:1397] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 11:19:57.709935 27452 tablet_bootstrap.cc:654] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:57.709935 27453 tablet_bootstrap.cc:654] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:57.711078 27452 log.cc:824] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:57.711387 27453 log.cc:824] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:57.714250 27452 tablet_bootstrap.cc:492] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: No bootstrap required, opened a new log
I20260812 11:19:57.714282 27453 tablet_bootstrap.cc:492] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: No bootstrap required, opened a new log
I20260812 11:19:57.714507 27452 ts_tablet_manager.cc:1397] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 11:19:57.714363 27450 raft_consensus.cc:348] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.714609 27453 ts_tablet_manager.cc:1397] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260812 11:19:57.714684 27450 raft_consensus.cc:374] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:57.714764 27450 raft_consensus.cc:729] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7303e4a6a4334176a2de3fb34bdf7d7e, State: Initialized, Role: FOLLOWER
I20260812 11:19:57.715380 27450 consensus_queue.cc:260] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.716591 27450 ts_tablet_manager.cc:1428] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 11:19:57.716974 27206 heartbeater.cc:503] Master 127.27.109.62:35209 was elected leader, sending a full tablet report...
I20260812 11:19:57.719503 27452 raft_consensus.cc:348] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.719861 27452 raft_consensus.cc:374] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:57.719925 27452 raft_consensus.cc:729] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f697a692e3e4467a9780b36216b8f98, State: Initialized, Role: FOLLOWER
I20260812 11:19:57.719801 27453 raft_consensus.cc:348] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.720345 27453 raft_consensus.cc:374] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:57.720450 27453 raft_consensus.cc:729] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e4b8e28ea84c5c808ec346f468fefb, State: Initialized, Role: FOLLOWER
I20260812 11:19:57.720472 27452 consensus_queue.cc:260] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.721159 27453 consensus_queue.cc:260] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.721901 27452 ts_tablet_manager.cc:1428] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 11:19:57.721959 26979 heartbeater.cc:503] Master 127.27.109.62:35209 was elected leader, sending a full tablet report...
I20260812 11:19:57.722548 27453 ts_tablet_manager.cc:1428] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Time spent starting tablet: real 0.008s user 0.005s sys 0.005s
I20260812 11:19:57.723117 27424 heartbeater.cc:503] Master 127.27.109.62:35209 was elected leader, sending a full tablet report...
W20260812 11:19:57.773864 27207 tablet.cc:2369] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:57.837705 27426 tablet.cc:2369] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:57.865669 27459 raft_consensus.cc:482] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:57.865958 27459 raft_consensus.cc:504] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.867465 27459 leader_election.cc:288] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f697a692e3e4467a9780b36216b8f98 (127.27.109.1:46573), 10e4b8e28ea84c5c808ec346f468fefb (127.27.109.3:39525)
I20260812 11:19:57.872936 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.868054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:36069 (local address 127.27.109.3:39525)
0812 11:19:57.868233 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.868240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.868592 (+ 352us) server_negotiation.cc:407] Connection header received
0812 11:19:57.868945 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.868982 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.869044 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.869140 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:57.869547 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.870082 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.871655 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.872148 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.872273 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.872323 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.872658 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.872785 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:19:57.873087 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.868085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:40569 (local address 127.27.109.1:46573)
0812 11:19:57.868229 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.868234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.868704 (+ 470us) server_negotiation.cc:407] Connection header received
0812 11:19:57.868803 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.868808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.868886 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.868986 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:57.869484 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.870110 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.871713 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.872325 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.872483 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.872555 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.872819 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.872923 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:19:57.872979 27463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.867851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39525 (local address 127.27.109.2:36069)
0812 11:19:57.868355 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:57.868377 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:57.868779 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:57.869197 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:57.869200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:57.869219 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:57.869371 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.869377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.870225 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.870232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.871489 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.871502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.872335 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.872340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.872473 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.872513 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 11:19:57.872622 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 11:19:57.872718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":28,"thread_start_us":98,"threads_started":1}
I20260812 11:19:57.873044 27464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.867998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46573 (local address 127.27.109.2:40569)
0812 11:19:57.868479 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:57.868494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:57.868689 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:57.869038 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:57.869042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:57.869063 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:57.869266 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.869273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.870358 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.870362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.871508 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:57.871520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.872503 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.872506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:57.872645 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.872679 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:19:57.872770 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:19:57.872868 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":61,"thread_start_us":165,"threads_started":1}
I20260812 11:19:57.874410 26911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916afe0454c346fc8c316f6454a430eb" candidate_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f697a692e3e4467a9780b36216b8f98" is_pre_election: true
I20260812 11:19:57.874413 27357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916afe0454c346fc8c316f6454a430eb" candidate_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e4b8e28ea84c5c808ec346f468fefb" is_pre_election: true
I20260812 11:19:57.874919 27357 raft_consensus.cc:2454] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7303e4a6a4334176a2de3fb34bdf7d7e in term 0.
I20260812 11:19:57.874919 26911 raft_consensus.cc:2454] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7303e4a6a4334176a2de3fb34bdf7d7e in term 0.
I20260812 11:19:57.875630 27015 leader_election.cc:302] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e4b8e28ea84c5c808ec346f468fefb, 7303e4a6a4334176a2de3fb34bdf7d7e; no voters:
I20260812 11:19:57.876060 27459 raft_consensus.cc:2781] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:57.876168 27459 raft_consensus.cc:482] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:57.876215 27459 raft_consensus.cc:3037] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:57.877434 27459 raft_consensus.cc:504] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.877849 27459 leader_election.cc:288] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [CANDIDATE]: Term 1 election: Requested vote from peers 9f697a692e3e4467a9780b36216b8f98 (127.27.109.1:46573), 10e4b8e28ea84c5c808ec346f468fefb (127.27.109.3:39525)
I20260812 11:19:57.878551 26911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916afe0454c346fc8c316f6454a430eb" candidate_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f697a692e3e4467a9780b36216b8f98"
I20260812 11:19:57.878798 26911 raft_consensus.cc:3037] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:57.879125 27357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916afe0454c346fc8c316f6454a430eb" candidate_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e4b8e28ea84c5c808ec346f468fefb"
I20260812 11:19:57.879367 27357 raft_consensus.cc:3037] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:57.880564 26911 raft_consensus.cc:2454] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7303e4a6a4334176a2de3fb34bdf7d7e in term 1.
I20260812 11:19:57.880807 27357 raft_consensus.cc:2454] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7303e4a6a4334176a2de3fb34bdf7d7e in term 1.
I20260812 11:19:57.881129 27014 leader_election.cc:302] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7303e4a6a4334176a2de3fb34bdf7d7e, 9f697a692e3e4467a9780b36216b8f98; no voters:
I20260812 11:19:57.881474 27459 raft_consensus.cc:2781] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:57.881739 27459 raft_consensus.cc:686] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 1 LEADER]: Becoming Leader. State: Replica: 7303e4a6a4334176a2de3fb34bdf7d7e, State: Running, Role: LEADER
I20260812 11:19:57.882150 27459 consensus_queue.cc:237] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } }
I20260812 11:19:57.886516 26670 catalog_manager.cc:5495] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7303e4a6a4334176a2de3fb34bdf7d7e (127.27.109.2). New cstate: current_term: 1 leader_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7303e4a6a4334176a2de3fb34bdf7d7e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 } health_report { overall_health: UNKNOWN } } }
I20260812 11:19:57.935885 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:57.930614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.27.109.2:33603)
0812 11:19:57.930885 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:57.930890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:57.930914 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:19:57.931159 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:57.931163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:57.931247 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:57.931418 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:57.931756 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.932647 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.933783 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:57.934039 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:57.934172 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:57.934599 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:57.935354 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:19:57.935524 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:57.935574 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 11:19:57.935675 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 11:19:57.944733 27357 raft_consensus.cc:1261] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 1 FOLLOWER]: Refusing update from remote peer 7303e4a6a4334176a2de3fb34bdf7d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:57.944989 26911 raft_consensus.cc:1261] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 1 FOLLOWER]: Refusing update from remote peer 7303e4a6a4334176a2de3fb34bdf7d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:57.945757 27459 consensus_queue.cc:1035] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e4b8e28ea84c5c808ec346f468fefb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:57.946185 27466 consensus_queue.cc:1035] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f697a692e3e4467a9780b36216b8f98" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:19:57.947393 26980 tablet.cc:2369] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:57.957273 27470 mvcc.cc:205] Tried to move back new op lower bound from 7317641617169448960 to 7317641616928964608. Current Snapshot: MvccSnapshot[applied={T|T < 7317641617169448960}]
I20260812 11:19:57.958786 27471 mvcc.cc:205] Tried to move back new op lower bound from 7317641617169448960 to 7317641616928964608. Current Snapshot: MvccSnapshot[applied={T|T < 7317641617169448960}]
I20260812 11:19:57.960294 27472 mvcc.cc:205] Tried to move back new op lower bound from 7317641617169448960 to 7317641616928964608. Current Snapshot: MvccSnapshot[applied={T|T < 7317641617169448960}]
I20260812 11:19:57.970906 26670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:57.971227 26670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:57.975073 26670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45b006ba14b94763bd70d66e1d0cc7b7: Sending DeleteTablet for 3 replicas of tablet 916afe0454c346fc8c316f6454a430eb
I20260812 11:19:57.976722 26884 tablet_service.cc:1514] Processing DeleteTablet for tablet 916afe0454c346fc8c316f6454a430eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:57 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:43776
I20260812 11:19:57.977522 27488 tablet_replica.cc:330] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98: stopping tablet replica
I20260812 11:19:57.977401 27328 tablet_service.cc:1514] Processing DeleteTablet for tablet 916afe0454c346fc8c316f6454a430eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:57 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:50308
I20260812 11:19:57.977985 27488 raft_consensus.cc:2229] T 916afe0454c346fc8c316f6454a430eb P 9f697a692e3e4467a9780b36216b8f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:57.978087 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 26778
I20260812 11:19:57.979178 27094 tablet_service.cc:1514] Processing DeleteTablet for tablet 916afe0454c346fc8c316f6454a430eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:57 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:41282
I20260812 11:19:58.048764 27491 tablet_replica.cc:330] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: stopping tablet replica
I20260812 11:19:58.049383 27491 raft_consensus.cc:2229] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:19:58.049939 27491 raft_consensus.cc:2258] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:58.052855 27490 tablet_replica.cc:330] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: stopping tablet replica
I20260812 11:19:58.053536 27490 raft_consensus.cc:2229] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:58.053712 27491 ts_tablet_manager.cc:1905] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:58.054234 27490 raft_consensus.cc:2258] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:58.056933 27490 ts_tablet_manager.cc:1905] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:58.057008 27491 ts_tablet_manager.cc:1918] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:58.057133 27491 log.cc:1196] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/916afe0454c346fc8c316f6454a430eb
I20260812 11:19:58.057487 27491 ts_tablet_manager.cc:1939] T 916afe0454c346fc8c316f6454a430eb P 7303e4a6a4334176a2de3fb34bdf7d7e: Deleting consensus metadata
I20260812 11:19:58.059384 26651 catalog_manager.cc:4841] TS 7303e4a6a4334176a2de3fb34bdf7d7e (127.27.109.2:33603): tablet 916afe0454c346fc8c316f6454a430eb (table test-table [id=964f1538dc4f4b3cbdcb8eec4060a791]) successfully deleted
I20260812 11:19:58.063230 27490 ts_tablet_manager.cc:1918] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:58.063452 27490 log.cc:1196] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/916afe0454c346fc8c316f6454a430eb
W20260812 11:19:58.063748 26650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:46573 (error 108)
I20260812 11:19:58.063936 27490 ts_tablet_manager.cc:1939] T 916afe0454c346fc8c316f6454a430eb P 10e4b8e28ea84c5c808ec346f468fefb: Deleting consensus metadata
I20260812 11:19:58.064648 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 26984
I20260812 11:19:58.077183 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:58.076646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46573 (local address 127.0.0.1:43786)
0812 11:19:58.076864 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:58.076997 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 11:19:58.077378 26651 catalog_manager.cc:4841] TS 10e4b8e28ea84c5c808ec346f468fefb (127.27.109.3:39525): tablet 916afe0454c346fc8c316f6454a430eb (table test-table [id=964f1538dc4f4b3cbdcb8eec4060a791]) successfully deleted
W20260812 11:19:58.077502 26650 catalog_manager.cc:4584] TS 9f697a692e3e4467a9780b36216b8f98 (127.27.109.1:46573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 916afe0454c346fc8c316f6454a430eb: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46573: connect: Connection refused (error 111)
I20260812 11:19:58.084437 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 27216
I20260812 11:19:58.112728 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 26624
2026-08-12T11:19:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3708 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T11:19:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:19:58Z Disabled control of system clock
I20260812 11:19:58.183408 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43221
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:19:58.334043 27502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:58.334678 27502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:58.334744 27502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:58.340181 27502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:19:58.340294 27502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:58.340318 27502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:58.340337 27502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:19:58.340360 27502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:19:58.346117 27502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:43221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:58.347435 27502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:58.348989 27502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:58.359746 27509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:58.359818 27515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:58.359776 27512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:58.360391 27502 server_base.cc:1034] running on GCE node
I20260812 11:19:58.361209 27502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:58.362661 27502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:58.364030 27502 hybrid_clock.cc:648] HybridClock initialized: now 1786533598363977 us; error 72 us; skew 500 ppm
I20260812 11:19:58.367003 27502 webserver.cc:460] Webserver started at http://127.27.109.62:43047/ using document root <none> and password file <none>
I20260812 11:19:58.367760 27502 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:58.367888 27502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:58.368134 27502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:58.370527 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "2b3df281669c8af6eb6d093d756a6908"
server_key_iv: "908f71d6b1c213e5756cb33f299d0cc8"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.371199 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "2b3df281669c8af6eb6d093d756a6908"
server_key_iv: "908f71d6b1c213e5756cb33f299d0cc8"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.375837 27502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:19:58.379117 27529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:58.380811 27502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:19:58.381058 27502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "2b3df281669c8af6eb6d093d756a6908"
server_key_iv: "908f71d6b1c213e5756cb33f299d0cc8"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.381229 27502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:58.398761 27502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:58.399703 27502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:58.399950 27502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:58.410048 27606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:43221 every 8 connection(s)
I20260812 11:19:58.410073 27502 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:43221
I20260812 11:19:58.412215 27502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:19:58.415596 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 27502
I20260812 11:19:58.415812 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:19:58.416231 28084 external_mini_cluster.cc:1428] Setting key 0117d8ab4cb6a0dcc14723175f404322
I20260812 11:19:58.417878 27607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:58.427187 27607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9: Bootstrap starting.
I20260812 11:19:58.430317 27607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:58.431216 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:58.419038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.27.109.62:43221)
0812 11:19:58.420108 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:58.420134 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:58.420186 (+ 52us) server_negotiation.cc:407] Connection header received
0812 11:19:58.421156 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:58.421184 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:58.421821 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:58.422299 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:58.422914 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:58.424568 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:58.425999 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:58.426787 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:58.427037 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:58.428600 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:58.428607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:58.428628 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:58.429221 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:58.429321 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:58.429354 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:58.429491 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:58.429764 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 11:19:58.430089 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":244,"threads_started":1}
I20260812 11:19:58.431409 27607 log.cc:824] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:58.434909 27607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9: No bootstrap required, opened a new log
I20260812 11:19:58.440024 27607 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43221 } }
I20260812 11:19:58.440428 27607 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:58.440510 27607 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f1d0b8c9774e948b8dd55ae54a7ab9, State: Initialized, Role: FOLLOWER
I20260812 11:19:58.441200 27607 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43221 } }
I20260812 11:19:58.441376 27607 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:19:58.441488 27607 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:19:58.441625 27607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:58.443079 27607 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43221 } }
I20260812 11:19:58.443595 27607 leader_election.cc:302] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f1d0b8c9774e948b8dd55ae54a7ab9; no voters:
I20260812 11:19:58.444087 27607 leader_election.cc:288] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:19:58.444341 27614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:58.444805 27614 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [term 1 LEADER]: Becoming Leader. State: Replica: d5f1d0b8c9774e948b8dd55ae54a7ab9, State: Running, Role: LEADER
I20260812 11:19:58.445258 27614 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43221 } }
I20260812 11:19:58.445945 27607 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:19:58.447522 27618 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43221 } } }
I20260812 11:19:58.447724 27618 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [sys.catalog]: This master's current role is: LEADER
I20260812 11:19:58.448290 27619 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f1d0b8c9774e948b8dd55ae54a7ab9. Latest consensus state: current_term: 1 leader_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f1d0b8c9774e948b8dd55ae54a7ab9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43221 } } }
I20260812 11:19:58.449286 27619 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9 [sys.catalog]: This master's current role is: LEADER
W20260812 11:19:58.452210 27642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:19:58.452343 27642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:19:58.452463 27632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:19:58.453248 27632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:19:58.460227 27632 catalog_manager.cc:1347] Generated new cluster ID: 0d2ff81b3df24dc5808fab2446af6fa0
I20260812 11:19:58.460340 27632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:19:58.494999 27632 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:19:58.497370 27632 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:19:58.508286 27632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9: Generated new TSK 0
I20260812 11:19:58.509693 27632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:19:58.518580 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43221
--builtin_ntp_servers=127.27.109.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:19:58.668583 27649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:58.669239 27649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:58.669304 27649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:58.674777 27649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:58.674934 27649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:58.675009 27649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260812 11:19:58.681123 27649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:58.682899 27649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:58.685359 27649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:58.700925 27656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:58.700845 27655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:58.700902 27658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:58.701423 27649 server_base.cc:1034] running on GCE node
I20260812 11:19:58.702044 27649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:58.703392 27649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:58.707010 27649 hybrid_clock.cc:648] HybridClock initialized: now 1786533598706811 us; error 203 us; skew 500 ppm
I20260812 11:19:58.711717 27649 webserver.cc:460] Webserver started at http://127.27.109.1:44515/ using document root <none> and password file <none>
I20260812 11:19:58.712751 27649 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:58.712909 27649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:58.713191 27649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:58.715914 27649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data/instance:
uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "b837b3b9f1f2bf2ffeb18fc95e678866"
server_key_iv: "a5368e53726c1e533d1451d0cad05951"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.716635 27649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "b837b3b9f1f2bf2ffeb18fc95e678866"
server_key_iv: "a5368e53726c1e533d1451d0cad05951"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.722625 27649 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:19:58.727958 27670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:58.730320 27649 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:19:58.730548 27649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/wal
uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "b837b3b9f1f2bf2ffeb18fc95e678866"
server_key_iv: "a5368e53726c1e533d1451d0cad05951"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.730731 27649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:58.745095 27649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:58.746464 27649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:58.746712 27649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:58.747644 27649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:58.748907 27649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:58.748968 27649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:58.749025 27649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:58.749042 27649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:58.765496 27649 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:42087
I20260812 11:19:58.765550 27846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:42087 every 8 connection(s)
I20260812 11:19:58.766765 27649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/data/info.pb
I20260812 11:19:58.771428 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 27649
I20260812 11:19:58.771626 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-0/wal/instance
I20260812 11:19:58.772921 28084 external_mini_cluster.cc:1428] Setting key 921d9993dbd89505d49ba5e3744da24c
I20260812 11:19:58.776844 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43221
--builtin_ntp_servers=127.27.109.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:19:58.786513 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:58.772276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50359 (local address 127.27.109.62:43221)
0812 11:19:58.772959 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:58.772967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:58.774359 (+ 1392us) server_negotiation.cc:407] Connection header received
0812 11:19:58.776720 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:58.776725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:58.776811 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:58.776943 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:58.778784 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:58.779738 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:58.782557 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:58.783169 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:58.783390 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:58.785618 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:58.785624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:58.785626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:58.785684 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:58.785730 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:58.785747 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:58.785858 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:58.786114 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 11:19:58.786256 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458}
I20260812 11:19:58.788515 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:58.772801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43221 (local address 127.27.109.1:50359)
0812 11:19:58.774091 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:58.774182 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:58.776301 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:58.777262 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:58.777276 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:58.777421 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:58.778350 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:58.778381 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:58.780337 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:58.780342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:58.781935 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:58.781964 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:58.783512 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:58.783520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:58.783806 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:58.784813 (+ 1007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:58.784816 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:58.784892 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:58.785020 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:58.785885 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:58.785901 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:58.785917 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 11:19:58.786115 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 11:19:58.786691 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":260,"threads_started":1}
I20260812 11:19:58.791468 27848 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43221
I20260812 11:19:58.792218 27848 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:58.793573 27848 heartbeater.cc:511] Master 127.27.109.62:43221 requested a full tablet report, sending...
I20260812 11:19:58.796943 27554 ts_manager.cc:194] Registered new tserver with Master: 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 (127.27.109.1:42087)
I20260812 11:19:58.799377 27554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:50359
W20260812 11:19:58.920720 27853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:58.921200 27853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:58.921299 27853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:58.926355 27853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:58.926479 27853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:58.926582 27853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260812 11:19:58.934185 27853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:58.936085 27853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:58.938171 27853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:58.952644 27867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:58.952642 27864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:58.952642 27865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:58.952698 27853 server_base.cc:1034] running on GCE node
I20260812 11:19:58.954253 27853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:58.955513 27853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:58.956913 27853 hybrid_clock.cc:648] HybridClock initialized: now 1786533598956860 us; error 74 us; skew 500 ppm
I20260812 11:19:58.960162 27853 webserver.cc:460] Webserver started at http://127.27.109.2:34689/ using document root <none> and password file <none>
I20260812 11:19:58.961138 27853 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:58.961247 27853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:58.961576 27853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:58.964805 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data/instance:
uuid: "3290aa1c29af499098890d1ddefd4b05"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "8648bda325aa49ca5b437dc85b249fa1"
server_key_iv: "ddf60af2e220a79533b791d8d076c036"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.965713 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "3290aa1c29af499098890d1ddefd4b05"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "8648bda325aa49ca5b437dc85b249fa1"
server_key_iv: "ddf60af2e220a79533b791d8d076c036"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.971836 27853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:19:58.976643 27875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:58.979357 27853 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:19:58.979699 27853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal
uuid: "3290aa1c29af499098890d1ddefd4b05"
format_stamp: "Formatted at 2026-08-12 11:19:58 on dist-test-slave-7z32"
server_key: "8648bda325aa49ca5b437dc85b249fa1"
server_key_iv: "ddf60af2e220a79533b791d8d076c036"
server_key_version: "encryptionkey@0"
I20260812 11:19:58.979929 27853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:59.005415 27853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:59.006577 27853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:59.006973 27853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:59.007876 27853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:59.009387 27853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:59.009485 27853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:59.009573 27853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:59.009615 27853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:59.027346 27853 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43775
I20260812 11:19:59.027410 28066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43775 every 8 connection(s)
I20260812 11:19:59.029292 27853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/data/info.pb
I20260812 11:19:59.038424 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 27853
I20260812 11:19:59.038640 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal/instance
I20260812 11:19:59.039065 28084 external_mini_cluster.cc:1428] Setting key ac6297890f8063e0716957e2710eb58b
I20260812 11:19:59.042351 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43221
--builtin_ntp_servers=127.27.109.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:19:59.047838 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.033310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:38335 (local address 127.27.109.62:43221)
0812 11:19:59.033600 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.033605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.036155 (+ 2550us) server_negotiation.cc:407] Connection header received
0812 11:19:59.038657 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.038664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.038770 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.039060 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:59.041116 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.041953 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.044155 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.044546 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.044661 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.046918 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:59.046926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:59.046932 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:59.046958 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:59.047021 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:59.047037 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:59.047161 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.047487 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.047591 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 11:19:59.049474 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.033693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43221 (local address 127.27.109.2:38335)
0812 11:19:59.035728 (+ 2035us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.035813 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.038118 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.039513 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.039534 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.039737 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:59.040830 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.040861 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.042222 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.042226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.043914 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.043935 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.044848 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.044856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.045203 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.046301 (+ 1098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:59.046304 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:59.046425 (+ 121us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:59.046593 (+ 168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:59.047203 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:59.047215 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:59.047222 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.047434 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.048003 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":216,"threads_started":1}
I20260812 11:19:59.051633 28067 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43221
I20260812 11:19:59.052052 28067 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:59.052901 28067 heartbeater.cc:511] Master 127.27.109.62:43221 requested a full tablet report, sending...
I20260812 11:19:59.055001 27554 ts_manager.cc:194] Registered new tserver with Master: 3290aa1c29af499098890d1ddefd4b05 (127.27.109.2:43775)
I20260812 11:19:59.055835 27554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:38335
W20260812 11:19:59.181936 28071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:19:59.182686 28071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:19:59.182790 28071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:19:59.188066 28071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:19:59.188172 28071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:19:59.188235 28071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260812 11:19:59.197674 28071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:19:59.199314 28071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:19:59.201143 28071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:19:59.215790 28077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:59.215785 28076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:19:59.215905 28080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:19:59.216579 28071 server_base.cc:1034] running on GCE node
I20260812 11:19:59.217168 28071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:19:59.218091 28071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:19:59.221581 28071 hybrid_clock.cc:648] HybridClock initialized: now 1786533599221454 us; error 42 us; skew 500 ppm
I20260812 11:19:59.225221 28071 webserver.cc:460] Webserver started at http://127.27.109.3:44095/ using document root <none> and password file <none>
I20260812 11:19:59.226189 28071 fs_manager.cc:359] Metadata directory not provided
I20260812 11:19:59.226338 28071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:19:59.226698 28071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:19:59.230300 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data/instance:
uuid: "fde4cb04cef1465c8f380d04c7a28cfa"
format_stamp: "Formatted at 2026-08-12 11:19:59 on dist-test-slave-7z32"
server_key: "1f8b9816b168371b32d6e82237d9f579"
server_key_iv: "640bf3b3b2bbbc00497d1d3653c573b4"
server_key_version: "encryptionkey@0"
I20260812 11:19:59.231344 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "fde4cb04cef1465c8f380d04c7a28cfa"
format_stamp: "Formatted at 2026-08-12 11:19:59 on dist-test-slave-7z32"
server_key: "1f8b9816b168371b32d6e82237d9f579"
server_key_iv: "640bf3b3b2bbbc00497d1d3653c573b4"
server_key_version: "encryptionkey@0"
I20260812 11:19:59.240021 28071 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260812 11:19:59.245730 28099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:59.248348 28071 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:19:59.248759 28071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal
uuid: "fde4cb04cef1465c8f380d04c7a28cfa"
format_stamp: "Formatted at 2026-08-12 11:19:59 on dist-test-slave-7z32"
server_key: "1f8b9816b168371b32d6e82237d9f579"
server_key_iv: "640bf3b3b2bbbc00497d1d3653c573b4"
server_key_version: "encryptionkey@0"
I20260812 11:19:59.249048 28071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:19:59.268622 28071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:19:59.269855 28071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:19:59.270185 28071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:19:59.271170 28071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:19:59.272815 28071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:19:59.272938 28071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:59.273031 28071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:19:59.273083 28071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:19:59.288542 28071 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35857
I20260812 11:19:59.288666 28275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35857 every 8 connection(s)
I20260812 11:19:59.290375 28071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/data/info.pb
I20260812 11:19:59.295393 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28071
I20260812 11:19:59.295580 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal/instance
I20260812 11:19:59.295967 28084 external_mini_cluster.cc:1428] Setting key 35a1b23c9b421d3118fcc2081df3df53
I20260812 11:19:59.305555 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.293559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:47759 (local address 127.27.109.62:43221)
0812 11:19:59.293730 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.293734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.295751 (+ 2017us) server_negotiation.cc:407] Connection header received
0812 11:19:59.297997 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.298004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.298136 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.298300 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:59.300094 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.300719 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.302558 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.302908 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.303050 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.304856 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:59.304861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:59.304863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:59.304884 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:59.304911 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:59.304921 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:59.305016 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.305234 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.305344 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:19:59.307158 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.294124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43221 (local address 127.27.109.3:47759)
0812 11:19:59.295409 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.295511 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.297660 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.298683 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.298696 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.298932 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:19:59.299860 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.299884 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.300929 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.300933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.302330 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.302349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.303133 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.303138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.303387 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.304286 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:19:59.304287 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:19:59.304354 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:19:59.304468 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:19:59.305034 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:19:59.305046 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:19:59.305055 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.305241 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.305728 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":231,"threads_started":1}
I20260812 11:19:59.309154 28276 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43221
I20260812 11:19:59.309571 28276 heartbeater.cc:464] Registering TS with master...
I20260812 11:19:59.310530 28276 heartbeater.cc:511] Master 127.27.109.62:43221 requested a full tablet report, sending...
I20260812 11:19:59.311976 27554 ts_manager.cc:194] Registered new tserver with Master: fde4cb04cef1465c8f380d04c7a28cfa (127.27.109.3:35857)
I20260812 11:19:59.312990 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:19:59.313000 27554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:47759
I20260812 11:19:59.322359 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.316552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.27.109.62:43221)
0812 11:19:59.316757 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.316762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.316969 (+ 207us) server_negotiation.cc:407] Connection header received
0812 11:19:59.317134 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.317137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.317202 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.317330 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:19:59.317836 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.318460 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.319858 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.320190 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.320338 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.321564 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:19:59.321568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:19:59.321571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:19:59.321593 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:19:59.321626 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:19:59.321646 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:19:59.321741 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.322068 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.322176 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 11:19:59.328181 27554 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:19:59.332356 27554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:19:59.358662 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.348945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:35857 (local address 127.0.0.1:58802)
0812 11:19:59.350602 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.350617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.350737 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.351830 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.351833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.351856 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:59.352169 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.352183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.353723 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.353729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.355967 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.355985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.357853 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.357857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.358008 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.358058 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.358210 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.358444 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":116,"threads_started":1}
I20260812 11:19:59.358668 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.349850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.27.109.3:35857)
0812 11:19:59.350598 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.350613 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.350740 (+ 127us) server_negotiation.cc:407] Connection header received
0812 11:19:59.351132 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.351140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.351407 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.351612 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:59.352348 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.353571 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.356200 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.357571 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.357749 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.358064 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.358296 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.358444 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":298,"threads_started":1}
I20260812 11:19:59.359361 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.348779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:42087 (local address 127.0.0.1:33074)
0812 11:19:59.349839 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.349906 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.350195 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.351388 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.351397 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.351430 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:59.351844 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.351861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.353619 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.353626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.355808 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.355824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.358270 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.358275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.358516 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.358565 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.358648 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.359146 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":118,"thread_start_us":130,"threads_started":1}
I20260812 11:19:59.359470 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.352180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.27.109.2:43775)
0812 11:19:59.352744 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.352757 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.352786 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:19:59.352881 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.352887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.353132 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.353293 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:59.353845 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.355158 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.356793 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.357751 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.357897 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.358153 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.358756 (+ 603us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.359284 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":133,"threads_started":1}
I20260812 11:19:59.359885 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.351378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43775 (local address 127.0.0.1:49936)
0812 11:19:59.352012 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.352031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.352137 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.353509 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.353512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.353543 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:59.353684 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.353693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.355383 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.355389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.356600 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.356612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.358270 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.358275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.358522 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.358566 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.358648 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.359692 (+ 1044us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":1022,"thread_start_us":231,"threads_started":1}
I20260812 11:19:59.360095 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.349068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.27.109.1:42087)
0812 11:19:59.350051 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.350062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.350100 (+ 38us) server_negotiation.cc:407] Connection header received
0812 11:19:59.350544 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.350553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.351115 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.351392 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:59.352051 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.353451 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.356018 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.357758 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.357897 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.358153 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.359763 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.359910 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":237,"threads_started":1}
I20260812 11:19:59.363487 28171 tablet_service.cc:1467] Processing CreateTablet for tablet 2a97080309f042ab843aef43537e10bc (DEFAULT_TABLE table=test-table [id=065cee9c24dd474faabc779a98599068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:59.364215 27745 tablet_service.cc:1467] Processing CreateTablet for tablet 2a97080309f042ab843aef43537e10bc (DEFAULT_TABLE table=test-table [id=065cee9c24dd474faabc779a98599068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:59.365378 28171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a97080309f042ab843aef43537e10bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:59.365760 27745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a97080309f042ab843aef43537e10bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:59.366137 27957 tablet_service.cc:1467] Processing CreateTablet for tablet 2a97080309f042ab843aef43537e10bc (DEFAULT_TABLE table=test-table [id=065cee9c24dd474faabc779a98599068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:19:59.367625 27957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a97080309f042ab843aef43537e10bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:19:59.374806 28303 tablet_bootstrap.cc:492] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79: Bootstrap starting.
I20260812 11:19:59.376833 28304 tablet_bootstrap.cc:492] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Bootstrap starting.
I20260812 11:19:59.376982 28305 tablet_bootstrap.cc:492] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Bootstrap starting.
I20260812 11:19:59.379706 28303 tablet_bootstrap.cc:654] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:59.379812 28304 tablet_bootstrap.cc:654] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:59.379812 28305 tablet_bootstrap.cc:654] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Neither blocks nor log segments found. Creating new log.
I20260812 11:19:59.380802 28303 log.cc:824] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:59.380934 28305 log.cc:824] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:59.380934 28304 log.cc:824] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Log is configured to *not* fsync() on all Append() calls
I20260812 11:19:59.384030 28305 tablet_bootstrap.cc:492] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: No bootstrap required, opened a new log
I20260812 11:19:59.384030 28303 tablet_bootstrap.cc:492] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79: No bootstrap required, opened a new log
I20260812 11:19:59.384315 28305 ts_tablet_manager.cc:1397] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 11:19:59.384385 28304 tablet_bootstrap.cc:492] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: No bootstrap required, opened a new log
I20260812 11:19:59.384411 28303 ts_tablet_manager.cc:1397] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 11:19:59.384814 28304 ts_tablet_manager.cc:1397] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:19:59.390095 28304 raft_consensus.cc:348] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.390864 28304 raft_consensus.cc:374] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:59.390974 28304 raft_consensus.cc:729] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde4cb04cef1465c8f380d04c7a28cfa, State: Initialized, Role: FOLLOWER
I20260812 11:19:59.391129 28305 raft_consensus.cc:348] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.391129 28303 raft_consensus.cc:348] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.391589 28305 raft_consensus.cc:374] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:59.391589 28303 raft_consensus.cc:374] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:19:59.391671 28303 raft_consensus.cc:729] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8f6d8d1c3d442fb7fbd72f3b7f8a79, State: Initialized, Role: FOLLOWER
I20260812 11:19:59.391688 28305 raft_consensus.cc:729] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3290aa1c29af499098890d1ddefd4b05, State: Initialized, Role: FOLLOWER
I20260812 11:19:59.391888 28304 consensus_queue.cc:260] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.392483 28305 consensus_queue.cc:260] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.392483 28303 consensus_queue.cc:260] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.393747 28304 ts_tablet_manager.cc:1428] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260812 11:19:59.393872 28305 ts_tablet_manager.cc:1428] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260812 11:19:59.393872 28303 ts_tablet_manager.cc:1428] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260812 11:19:59.394343 27848 heartbeater.cc:503] Master 127.27.109.62:43221 was elected leader, sending a full tablet report...
I20260812 11:19:59.394343 28067 heartbeater.cc:503] Master 127.27.109.62:43221 was elected leader, sending a full tablet report...
I20260812 11:19:59.394336 28276 heartbeater.cc:503] Master 127.27.109.62:43221 was elected leader, sending a full tablet report...
W20260812 11:19:59.521792 27849 tablet.cc:2369] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:59.532261 28068 tablet.cc:2369] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:19:59.542112 28277 tablet.cc:2369] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:19:59.659754 28313 raft_consensus.cc:482] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:59.660118 28313 raft_consensus.cc:504] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.662184 28310 raft_consensus.cc:482] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:19:59.662518 28310 raft_consensus.cc:504] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.662609 28313 leader_election.cc:288] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde4cb04cef1465c8f380d04c7a28cfa (127.27.109.3:35857), 3290aa1c29af499098890d1ddefd4b05 (127.27.109.2:43775)
I20260812 11:19:59.664444 28310 leader_election.cc:288] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 (127.27.109.1:42087), 3290aa1c29af499098890d1ddefd4b05 (127.27.109.2:43775)
I20260812 11:19:59.669261 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.663120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57735 (local address 127.27.109.3:35857)
0812 11:19:59.663273 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.663278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.663626 (+ 348us) server_negotiation.cc:407] Connection header received
0812 11:19:59.663878 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.663883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.663969 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.664077 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:59.665146 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.665971 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.667890 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.668529 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.668648 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.668707 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.669041 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.669111 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:19:59.669598 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.664976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38835 (local address 127.27.109.1:42087)
0812 11:19:59.665093 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.665098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.665110 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:19:59.665154 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.665157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.665200 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.665279 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:59.665745 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.666443 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.668006 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.668563 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.668666 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.668706 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.669310 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.669468 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":52}
I20260812 11:19:59.669261 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.664682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:42087 (local address 127.27.109.3:38835)
0812 11:19:59.664847 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.664865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.664951 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.665420 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.665424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.665450 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:59.665600 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.665611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.666588 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.666593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.667875 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.667886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.668763 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.668768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.668896 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.668919 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.668996 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.669084 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 11:19:59.669824 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.662495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:35857 (local address 127.27.109.1:57735)
0812 11:19:59.663492 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.663512 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.663711 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.664234 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.664237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.664262 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:59.664454 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.664466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.666132 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.666136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.667721 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.667733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.668806 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.668809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.668901 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.668920 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.669584 (+ 664us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.669666 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":573,"thread_start_us":121,"threads_started":1}
I20260812 11:19:59.670091 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.662495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43775 (local address 127.27.109.1:35569)
0812 11:19:59.663277 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.663301 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.663533 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.664052 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.664055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.664075 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:59.664237 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.664243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.666326 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.666332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.667916 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.667929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.669189 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.669194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.669314 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.669382 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.669645 (+ 263us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.669773 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":3,"thread_start_us":153,"threads_started":1}
I20260812 11:19:59.670459 28200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" is_pre_election: true
I20260812 11:19:59.670460 27776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" is_pre_election: true
I20260812 11:19:59.670650 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.662785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:35569 (local address 127.27.109.2:43775)
0812 11:19:59.662975 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.662983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.663479 (+ 496us) server_negotiation.cc:407] Connection header received
0812 11:19:59.663833 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.663836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.663902 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.664004 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:59.664465 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.665194 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.668094 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.668823 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.668941 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.669005 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.669558 (+ 553us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.670484 (+ 926us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":796,"server-negotiator.queue_time_us":93}
I20260812 11:19:59.670758 28200 raft_consensus.cc:2454] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 in term 0.
I20260812 11:19:59.670758 27776 raft_consensus.cc:2454] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde4cb04cef1465c8f380d04c7a28cfa in term 0.
I20260812 11:19:59.671347 28105 leader_election.cc:302] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8f6d8d1c3d442fb7fbd72f3b7f8a79, fde4cb04cef1465c8f380d04c7a28cfa; no voters:
I20260812 11:19:59.671600 27673 leader_election.cc:302] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8f6d8d1c3d442fb7fbd72f3b7f8a79, fde4cb04cef1465c8f380d04c7a28cfa; no voters:
I20260812 11:19:59.671716 28310 raft_consensus.cc:2781] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:59.671819 28310 raft_consensus.cc:482] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:59.671741 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3290aa1c29af499098890d1ddefd4b05" is_pre_election: true
I20260812 11:19:59.671869 28310 raft_consensus.cc:3037] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:59.672111 27989 raft_consensus.cc:2454] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 in term 0.
I20260812 11:19:59.672103 28313 raft_consensus.cc:2781] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:19:59.672272 28313 raft_consensus.cc:482] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:19:59.672339 28313 raft_consensus.cc:3037] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:59.672889 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.664912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43775 (local address 127.27.109.3:51151)
0812 11:19:59.665533 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.665554 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:19:59.665644 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:19:59.666167 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:19:59.666171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:19:59.666191 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:19:59.666363 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.666375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.667681 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.667685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.669090 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:19:59.669107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.672068 (+ 2961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.672072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:19:59.672371 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.672457 (+ 86us) client_negotiation.cc:769] Sending connection context
0812 11:19:59.672544 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 11:19:59.672691 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":175,"threads_started":1}
I20260812 11:19:59.673403 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.665463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:51151 (local address 127.27.109.2:43775)
0812 11:19:59.665880 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.665885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.665900 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:19:59.665957 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.665960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.666040 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.666151 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:19:59.666546 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.667521 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.669249 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.671564 (+ 2315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.672960 (+ 1396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.673035 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.673085 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.673190 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":128,"threads_started":1}
I20260812 11:19:59.673491 28310 raft_consensus.cc:504] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.673981 28310 leader_election.cc:288] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [CANDIDATE]: Term 1 election: Requested vote from peers 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 (127.27.109.1:42087), 3290aa1c29af499098890d1ddefd4b05 (127.27.109.2:43775)
I20260812 11:19:59.674104 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3290aa1c29af499098890d1ddefd4b05" is_pre_election: true
I20260812 11:19:59.674293 27989 raft_consensus.cc:2454] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde4cb04cef1465c8f380d04c7a28cfa in term 0.
I20260812 11:19:59.674307 27776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79"
I20260812 11:19:59.674278 28313 raft_consensus.cc:504] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.674551 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3290aa1c29af499098890d1ddefd4b05"
I20260812 11:19:59.674705 27988 raft_consensus.cc:3037] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:19:59.674906 28313 leader_election.cc:288] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [CANDIDATE]: Term 1 election: Requested vote from peers fde4cb04cef1465c8f380d04c7a28cfa (127.27.109.3:35857), 3290aa1c29af499098890d1ddefd4b05 (127.27.109.2:43775)
I20260812 11:19:59.674973 27776 raft_consensus.cc:2379] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fde4cb04cef1465c8f380d04c7a28cfa in current term 1: Already voted for candidate 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 in this term.
I20260812 11:19:59.675341 28200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde4cb04cef1465c8f380d04c7a28cfa"
I20260812 11:19:59.675551 28200 raft_consensus.cc:2379] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 in current term 1: Already voted for candidate fde4cb04cef1465c8f380d04c7a28cfa in this term.
I20260812 11:19:59.675511 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a97080309f042ab843aef43537e10bc" candidate_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3290aa1c29af499098890d1ddefd4b05"
I20260812 11:19:59.676196 27677 leader_election.cc:302] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d8f6d8d1c3d442fb7fbd72f3b7f8a79; no voters: 3290aa1c29af499098890d1ddefd4b05, fde4cb04cef1465c8f380d04c7a28cfa
I20260812 11:19:59.676561 28313 raft_consensus.cc:2726] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 11:19:59.676558 27988 raft_consensus.cc:2454] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde4cb04cef1465c8f380d04c7a28cfa in term 1.
I20260812 11:19:59.676920 28102 leader_election.cc:302] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3290aa1c29af499098890d1ddefd4b05, fde4cb04cef1465c8f380d04c7a28cfa; no voters: 5d8f6d8d1c3d442fb7fbd72f3b7f8a79
I20260812 11:19:59.677222 28310 raft_consensus.cc:2781] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:19:59.677574 28310 raft_consensus.cc:686] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 1 LEADER]: Becoming Leader. State: Replica: fde4cb04cef1465c8f380d04c7a28cfa, State: Running, Role: LEADER
I20260812 11:19:59.678010 28310 consensus_queue.cc:237] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } }
I20260812 11:19:59.682353 27551 catalog_manager.cc:5495] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa reported cstate change: term changed from 0 to 1, leader changed from <none> to fde4cb04cef1465c8f380d04c7a28cfa (127.27.109.3). New cstate: current_term: 1 leader_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fde4cb04cef1465c8f380d04c7a28cfa" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 } health_report { overall_health: UNKNOWN } } }
I20260812 11:19:59.742169 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:19:59.736436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.27.109.3:35857)
0812 11:19:59.736860 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0812 11:19:59.736866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:19:59.736891 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:19:59.737169 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:19:59.737172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:19:59.737255 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:19:59.737416 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:19:59.737924 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.738521 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.740066 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:19:59.740354 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:19:59.740475 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:19:59.741038 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:19:59.741580 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:19:59.741706 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:19:59.741954 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 11:19:59.742023 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260812 11:19:59.751149 27776 raft_consensus.cc:1261] T 2a97080309f042ab843aef43537e10bc P 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 [term 1 FOLLOWER]: Refusing update from remote peer fde4cb04cef1465c8f380d04c7a28cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:59.751175 27988 raft_consensus.cc:1261] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 1 FOLLOWER]: Refusing update from remote peer fde4cb04cef1465c8f380d04c7a28cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:19:59.752043 28319 consensus_queue.cc:1035] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d8f6d8d1c3d442fb7fbd72f3b7f8a79" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:59.752432 28310 consensus_queue.cc:1035] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3290aa1c29af499098890d1ddefd4b05" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:19:59.770017 28324 mvcc.cc:205] Tried to move back new op lower bound from 7317641624570028032 to 7317641624284729344. Current Snapshot: MvccSnapshot[applied={T|T < 7317641624570028032 or (T in {7317641624570028032})}]
I20260812 11:19:59.776342 27551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:59.776592 27551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:19:59.779973 27551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5f1d0b8c9774e948b8dd55ae54a7ab9: Sending DeleteTablet for 3 replicas of tablet 2a97080309f042ab843aef43537e10bc
I20260812 11:19:59.781327 28171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a97080309f042ab843aef43537e10bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:59 UTC) from {username='slave'} at 127.0.0.1:58802
I20260812 11:19:59.781556 27957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a97080309f042ab843aef43537e10bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:59 UTC) from {username='slave'} at 127.0.0.1:49936
I20260812 11:19:59.782275 28336 tablet_replica.cc:330] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: stopping tablet replica
I20260812 11:19:59.782763 27745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a97080309f042ab843aef43537e10bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:19:59 UTC) from {username='slave'} at 127.0.0.1:33074
I20260812 11:19:59.783245 28336 raft_consensus.cc:2229] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:19:59.784214 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 27649
I20260812 11:19:59.784868 28336 raft_consensus.cc:2258] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:59.837782 28337 tablet_replica.cc:330] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: stopping tablet replica
I20260812 11:19:59.839915 28337 raft_consensus.cc:2229] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:19:59.840692 28336 ts_tablet_manager.cc:1905] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:59.840787 28337 raft_consensus.cc:2258] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:19:59.843956 28337 ts_tablet_manager.cc:1905] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:19:59.846670 28336 ts_tablet_manager.cc:1918] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:59.846915 28336 log.cc:1196] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-2/wal/wals/2a97080309f042ab843aef43537e10bc
I20260812 11:19:59.847513 28336 ts_tablet_manager.cc:1939] T 2a97080309f042ab843aef43537e10bc P fde4cb04cef1465c8f380d04c7a28cfa: Deleting consensus metadata
I20260812 11:19:59.848800 28337 ts_tablet_manager.cc:1918] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:19:59.849059 28337 log.cc:1196] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Deleting WAL directory at /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533282802791-28084-0/minicluster-data/ts-1/wal/wals/2a97080309f042ab843aef43537e10bc
I20260812 11:19:59.849534 28337 ts_tablet_manager.cc:1939] T 2a97080309f042ab843aef43537e10bc P 3290aa1c29af499098890d1ddefd4b05: Deleting consensus metadata
W20260812 11:19:59.849648 27535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:42087 (error 108)
I20260812 11:19:59.850518 27531 catalog_manager.cc:4841] TS fde4cb04cef1465c8f380d04c7a28cfa (127.27.109.3:35857): tablet 2a97080309f042ab843aef43537e10bc (table test-table [id=065cee9c24dd474faabc779a98599068]) successfully deleted
W20260812 11:19:59.851477 28300 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:19:59.851025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:42087 (local address 127.0.0.1:33086)
0812 11:19:59.851207 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 11:19:59.851324 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:42087: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 11:19:59.851882 27535 catalog_manager.cc:4584] TS 5d8f6d8d1c3d442fb7fbd72f3b7f8a79 (127.27.109.1:42087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a97080309f042ab843aef43537e10bc: Network error: Client connection negotiation failed: client connection to 127.27.109.1:42087: connect: Connection reset by peer (error 104)
I20260812 11:19:59.851925 27534 catalog_manager.cc:4841] TS 3290aa1c29af499098890d1ddefd4b05 (127.27.109.2:43775): tablet 2a97080309f042ab843aef43537e10bc (table test-table [id=065cee9c24dd474faabc779a98599068]) successfully deleted
I20260812 11:19:59.852809 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 27853
I20260812 11:19:59.875756 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28071
I20260812 11:19:59.903091 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 27502
2026-08-12T11:19:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1768 ms)
[----------] 6 tests from EncryptionPolicyTest (16092 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:00 dist-test-slave-7z32 krb5kdc[28346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:00 dist-test-slave-7z32 krb5kdc[28346](info): set up 2 sockets
Aug 12 11:20:00 dist-test-slave-7z32 krb5kdc[28346](info): commencing operation
krb5kdc: starting...
W20260812 11:20:01.433339 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.497s user 0.004s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:01 dist-test-slave-7z32 krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:01.607412 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:01.761597 28369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:01.762142 28369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:01.762243 28369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:01.767113 28369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:01.767293 28369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:01.767334 28369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:01.767369 28369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:01.767406 28369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:01.775038 28369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:01.776801 28369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:01.778291 28369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:01.789741 28377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:01.789773 28378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:01.789741 28386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:01.790140 28369 server_base.cc:1034] running on GCE node
I20260812 11:20:01.791123 28369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:01.792544 28369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:01.793852 28369 hybrid_clock.cc:648] HybridClock initialized: now 1786533601793811 us; error 73 us; skew 500 ppm
Aug 12 11:20:01 dist-test-slave-7z32 krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:01.797364 28369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:01.798472 28369 webserver.cc:460] Webserver started at http://127.0.0.1:33451/ using document root <none> and password file <none>
I20260812 11:20:01.799240 28369 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:01.799320 28369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:01.799521 28369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:01.801844 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "422e8e43371e4f34825ad0d1b1894f82"
format_stamp: "Formatted at 2026-08-12 11:20:01 on dist-test-slave-7z32"
server_key: "d785e6229273dfc9879e9a6d35332e64"
server_key_iv: "9a81a1d0217c83e7bbc5bac6aad2d2dc"
server_key_version: "encryptionkey@0"
I20260812 11:20:01.802429 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "422e8e43371e4f34825ad0d1b1894f82"
format_stamp: "Formatted at 2026-08-12 11:20:01 on dist-test-slave-7z32"
server_key: "d785e6229273dfc9879e9a6d35332e64"
server_key_iv: "9a81a1d0217c83e7bbc5bac6aad2d2dc"
server_key_version: "encryptionkey@0"
I20260812 11:20:01.807251 28369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:20:01.810927 28396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:01.812840 28369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:20:01.813056 28369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "422e8e43371e4f34825ad0d1b1894f82"
format_stamp: "Formatted at 2026-08-12 11:20:01 on dist-test-slave-7z32"
server_key: "d785e6229273dfc9879e9a6d35332e64"
server_key_iv: "9a81a1d0217c83e7bbc5bac6aad2d2dc"
server_key_version: "encryptionkey@0"
I20260812 11:20:01.813205 28369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:01.826781 28369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:01.830742 28369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:01.831081 28369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:01.839789 28369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42257
I20260812 11:20:01.840018 28482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42257 every 8 connection(s)
I20260812 11:20:01.841092 28369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:01.845228 28487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:01.849895 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28369
I20260812 11:20:01.850060 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:01.850472 28084 external_mini_cluster.cc:1428] Setting key fdafcc08b859f5e3adb4b0471f19044e
I20260812 11:20:01.854033 28487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82: Bootstrap starting.
I20260812 11:20:01.857808 28487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:01.859076 28487 log.cc:824] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:01.862299 28487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82: No bootstrap required, opened a new log
Aug 12 11:20:01 dist-test-slave-7z32 krb5kdc[28346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:20:01.867818 28487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422e8e43371e4f34825ad0d1b1894f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42257 } }
I20260812 11:20:01.868240 28487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:01.868320 28487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422e8e43371e4f34825ad0d1b1894f82, State: Initialized, Role: FOLLOWER
I20260812 11:20:01.868999 28487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422e8e43371e4f34825ad0d1b1894f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42257 } }
I20260812 11:20:01.869210 28487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:01.869292 28487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:01.869402 28487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:01.871107 28487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422e8e43371e4f34825ad0d1b1894f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42257 } }
I20260812 11:20:01.871567 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:01.852700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40821 (local address 127.0.0.1:42257)
0812 11:20:01.853369 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:01.853393 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:01.853454 (+ 61us) server_negotiation.cc:407] Connection header received
0812 11:20:01.854520 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:01.854552 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:01.855140 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:01.855646 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:01.856547 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:01.858250 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:01.859511 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:01.860182 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:01.860415 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:01.863614 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:01.863626 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:01.863651 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:01.863698 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:01.866977 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:01.867741 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:01.867748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:01.867755 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:01.867938 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:01.868447 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:01.868451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:01.868453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:01.869092 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:01.869399 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:01.869908 (+ 509us) server_negotiation.cc:299] Negotiation successful
0812 11:20:01.870204 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":185,"threads_started":1}
I20260812 11:20:01.871585 28487 leader_election.cc:302] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 422e8e43371e4f34825ad0d1b1894f82; no voters:
I20260812 11:20:01.872161 28487 leader_election.cc:288] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:01.872500 28496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:01.873059 28496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [term 1 LEADER]: Becoming Leader. State: Replica: 422e8e43371e4f34825ad0d1b1894f82, State: Running, Role: LEADER
I20260812 11:20:01.873597 28496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422e8e43371e4f34825ad0d1b1894f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42257 } }
I20260812 11:20:01.873811 28487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:01.875901 28498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 422e8e43371e4f34825ad0d1b1894f82. Latest consensus state: current_term: 1 leader_uuid: "422e8e43371e4f34825ad0d1b1894f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422e8e43371e4f34825ad0d1b1894f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42257 } } }
I20260812 11:20:01.876098 28498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:01.876098 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:01.876364 28497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "422e8e43371e4f34825ad0d1b1894f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422e8e43371e4f34825ad0d1b1894f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42257 } } }
I20260812 11:20:01.876528 28497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:01.876820 28511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:01.881019 28511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:01.889273 28511 catalog_manager.cc:1347] Generated new cluster ID: 766966841bbd45028c6ba835a1908e3d
I20260812 11:20:01.889397 28511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:01.894225 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:01.879250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34149 (local address 127.0.0.1:42257)
0812 11:20:01.879374 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:01.879380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:01.883796 (+ 4416us) server_negotiation.cc:407] Connection header received
0812 11:20:01.884036 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:01.884040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:01.884097 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:01.884200 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:01.885132 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:01.885738 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:01.886896 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:01.887269 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:01.887390 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:01.889739 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:01.889745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:01.889748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:01.889781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:01.891874 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:01.892632 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:01.892636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:01.892638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:01.892758 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:01.893146 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:01.893149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:01.893151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:01.893397 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:01.893569 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:01.893886 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 11:20:01.894020 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:20:01.909701 28511 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:01.911111 28511 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:01.926736 28511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 422e8e43371e4f34825ad0d1b1894f82: Generated new TSK 0
I20260812 11:20:01.927942 28511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:01.936653 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28369
W20260812 11:20:01.954282 28340 connection.cc:535] client connection to 127.0.0.1:42257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42257): Connection reset by peer (error 104)
2026-08-12T11:20:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:02 dist-test-slave-7z32 krb5kdc[28530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:02 dist-test-slave-7z32 krb5kdc[28530](info): set up 2 sockets
Aug 12 11:20:02 dist-test-slave-7z32 krb5kdc[28530](info): commencing operation
krb5kdc: starting...
W20260812 11:20:03.348853 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.378s user 0.003s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:03 dist-test-slave-7z32 krb5kdc[28530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:03.515417 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:03.660712 28548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:03.661094 28548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:03.661145 28548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:03.666097 28548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:03.666194 28548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:03.666217 28548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:03.666237 28548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:03.666260 28548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:03.672904 28548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:03.674325 28548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:03.675637 28548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:03.686937 28557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:03.686962 28556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:03.686954 28548 server_base.cc:1034] running on GCE node
W20260812 11:20:03.686950 28559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:03.688053 28548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:03.689407 28548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:03.690732 28548 hybrid_clock.cc:648] HybridClock initialized: now 1786533603690692 us; error 70 us; skew 500 ppm
Aug 12 11:20:03 dist-test-slave-7z32 krb5kdc[28530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:03.703259 28548 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:03.704625 28548 webserver.cc:460] Webserver started at http://127.0.0.1:32889/ using document root <none> and password file <none>
I20260812 11:20:03.705435 28548 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:03.705531 28548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:03.705811 28548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:03.708492 28548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b"
format_stamp: "Formatted at 2026-08-12 11:20:03 on dist-test-slave-7z32"
server_key: "ebe1080453bd60a79efaae1b30b84943"
server_key_iv: "b35dd35ccb24fbd644f0f161dd01ce3b"
server_key_version: "encryptionkey@0"
I20260812 11:20:03.709225 28548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b"
format_stamp: "Formatted at 2026-08-12 11:20:03 on dist-test-slave-7z32"
server_key: "ebe1080453bd60a79efaae1b30b84943"
server_key_iv: "b35dd35ccb24fbd644f0f161dd01ce3b"
server_key_version: "encryptionkey@0"
I20260812 11:20:03.714448 28548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:20:03.718008 28577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:03.719973 28548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 11:20:03.720284 28548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b"
format_stamp: "Formatted at 2026-08-12 11:20:03 on dist-test-slave-7z32"
server_key: "ebe1080453bd60a79efaae1b30b84943"
server_key_iv: "b35dd35ccb24fbd644f0f161dd01ce3b"
server_key_version: "encryptionkey@0"
I20260812 11:20:03.720525 28548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:03.740181 28548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:03.744580 28548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:03.744887 28548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:03.754273 28548 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42491
I20260812 11:20:03.754405 28678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42491 every 8 connection(s)
I20260812 11:20:03.755542 28548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:03.756701 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28548
I20260812 11:20:03.756917 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:03.757264 28084 external_mini_cluster.cc:1428] Setting key c1cb222e79974a8db4d084311a926369
I20260812 11:20:03.760185 28681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:03.769279 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:03.759158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49819 (local address 127.0.0.1:42491)
0812 11:20:03.759671 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:03.759689 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:03.759733 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:20:03.760528 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:03.760550 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:03.760962 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:03.761313 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:03.761852 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:03.763233 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:03.764770 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:03.765350 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:03.765552 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:03.767064 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:03.767069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:03.767085 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:03.767611 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:03.767681 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:03.767707 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:03.767833 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:03.768115 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 11:20:03.768298 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":182,"threads_started":1}
I20260812 11:20:03.769582 28681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b: Bootstrap starting.
I20260812 11:20:03.773066 28681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:03.773124 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:03.774061 28681 log.cc:824] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:03.777240 28681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b: No bootstrap required, opened a new log
I20260812 11:20:03.781124 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:03.775853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43679 (local address 127.0.0.1:42491)
0812 11:20:03.776025 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:03.776031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:03.776355 (+ 324us) server_negotiation.cc:407] Connection header received
0812 11:20:03.776483 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:03.776485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:03.776537 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:03.776621 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:03.777015 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:03.777659 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:03.778945 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:03.779182 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:03.779285 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:03.780461 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:03.780464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:03.780466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:03.780481 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:03.780511 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:03.780522 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:03.780601 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:03.780851 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 11:20:03.780936 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:20:03.782792 28681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260812 11:20:03.783210 28681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:03.783300 28681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eca4ac7b08a4ec0b9b1a0cfd8900e0b, State: Initialized, Role: FOLLOWER
I20260812 11:20:03.783861 28681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260812 11:20:03.784040 28681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:03.784140 28681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:03.784286 28681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:03.785569 28681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260812 11:20:03.786029 28681 leader_election.cc:302] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eca4ac7b08a4ec0b9b1a0cfd8900e0b; no voters:
I20260812 11:20:03.786500 28681 leader_election.cc:288] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:03.786597 28699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:03.786926 28699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [term 1 LEADER]: Becoming Leader. State: Replica: 6eca4ac7b08a4ec0b9b1a0cfd8900e0b, State: Running, Role: LEADER
I20260812 11:20:03.787356 28699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260812 11:20:03.788456 28681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:03.789774 28701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } } }
I20260812 11:20:03.790045 28701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:03.790609 28700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eca4ac7b08a4ec0b9b1a0cfd8900e0b. Latest consensus state: current_term: 1 leader_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eca4ac7b08a4ec0b9b1a0cfd8900e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } } }
I20260812 11:20:03.790784 28700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:03.790663 28712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:03.795107 28712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:03.802008 28712 catalog_manager.cc:1347] Generated new cluster ID: fc917a47f9364d01a54be42afd6c368c
I20260812 11:20:03.802155 28712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:03.810313 28712 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:03.811931 28712 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:03.825701 28712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eca4ac7b08a4ec0b9b1a0cfd8900e0b: Generated new TSK 0
I20260812 11:20:03.826686 28712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:03.831617 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28548
W20260812 11:20:03.858403 28527 connection.cc:535] client connection to 127.0.0.1:42491 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42491): Connection reset by peer (error 104)
2026-08-12T11:20:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:03 dist-test-slave-7z32 krb5kdc[28737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:03 dist-test-slave-7z32 krb5kdc[28737](info): set up 2 sockets
Aug 12 11:20:03 dist-test-slave-7z32 krb5kdc[28737](info): commencing operation
krb5kdc: starting...
W20260812 11:20:05.269347 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.395s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:05 dist-test-slave-7z32 krb5kdc[28737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:05.439577 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:05.580958 28757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:05.581398 28757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:05.581506 28757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:05.586545 28757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:05.586715 28757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:05.586755 28757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:05.586803 28757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:05.586891 28757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:05.593415 28757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37873
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:05.594883 28757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:05.596158 28757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:05.608044 28773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:05.608163 28765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:05.608260 28757 server_base.cc:1034] running on GCE node
W20260812 11:20:05.608072 28766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:05.609488 28757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:05.611320 28757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:05.614748 28757 hybrid_clock.cc:648] HybridClock initialized: now 1786533605614649 us; error 96 us; skew 500 ppm
Aug 12 11:20:05 dist-test-slave-7z32 krb5kdc[28737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:05.619426 28757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:05.621090 28757 webserver.cc:460] Webserver started at http://127.0.0.1:41753/ using document root <none> and password file <none>
I20260812 11:20:05.622069 28757 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:05.622210 28757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:05.622566 28757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:05.625896 28757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "9b7fa366f73047f78897ec5cdfa75749"
format_stamp: "Formatted at 2026-08-12 11:20:05 on dist-test-slave-7z32"
server_key: "5689ee82c1aae57ef2964536c2ada00a"
server_key_iv: "f53c892136f0b8f0adf98ff71c5ca5dd"
server_key_version: "encryptionkey@0"
I20260812 11:20:05.626782 28757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "9b7fa366f73047f78897ec5cdfa75749"
format_stamp: "Formatted at 2026-08-12 11:20:05 on dist-test-slave-7z32"
server_key: "5689ee82c1aae57ef2964536c2ada00a"
server_key_iv: "f53c892136f0b8f0adf98ff71c5ca5dd"
server_key_version: "encryptionkey@0"
I20260812 11:20:05.633066 28757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 11:20:05.637184 28783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:05.639302 28757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:20:05.639603 28757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "9b7fa366f73047f78897ec5cdfa75749"
format_stamp: "Formatted at 2026-08-12 11:20:05 on dist-test-slave-7z32"
server_key: "5689ee82c1aae57ef2964536c2ada00a"
server_key_iv: "f53c892136f0b8f0adf98ff71c5ca5dd"
server_key_version: "encryptionkey@0"
I20260812 11:20:05.639807 28757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:05.648080 28757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:05.653438 28757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:05.653774 28757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:05.664278 28757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37873
I20260812 11:20:05.664314 28867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37873 every 8 connection(s)
I20260812 11:20:05.665676 28757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:05.669387 28868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:05.671350 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28757
I20260812 11:20:05.671512 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:05.672091 28084 external_mini_cluster.cc:1428] Setting key 7ca3c4a8eb80cf54d8bc6f1ce8878a20
I20260812 11:20:05.677314 28868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749: Bootstrap starting.
I20260812 11:20:05.679668 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:05.674207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60265 (local address 127.0.0.1:37873)
0812 11:20:05.674991 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:05.675017 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:05.675073 (+ 56us) server_negotiation.cc:407] Connection header received
0812 11:20:05.675977 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:05.676005 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:05.676442 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:05.676880 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:05.678031 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:05.678037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:05.678051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:05.678066 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:05.678144 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:05.678174 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:05.678358 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:05.678453 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 11:20:05.678627 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":235,"threads_started":1}
I20260812 11:20:05.680474 28868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:05.681416 28868 log.cc:824] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:05.684422 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:05.684703 28868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749: No bootstrap required, opened a new log
I20260812 11:20:05.690146 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:05.687726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54989 (local address 127.0.0.1:37873)
0812 11:20:05.687935 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:05.687945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:05.688072 (+ 127us) server_negotiation.cc:407] Connection header received
0812 11:20:05.688231 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:05.688236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:05.688304 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:05.688477 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:05.689490 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:05.689495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:05.689497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:05.689513 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:05.689552 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:05.689566 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:05.689712 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:05.689838 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 11:20:05.689894 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:20:05.689916 28868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7fa366f73047f78897ec5cdfa75749" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260812 11:20:05.690368 28868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:05.690483 28868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7fa366f73047f78897ec5cdfa75749, State: Initialized, Role: FOLLOWER
I20260812 11:20:05.691186 28868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7fa366f73047f78897ec5cdfa75749" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260812 11:20:05.691418 28868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:05.691500 28868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:05.691613 28868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:05.692840 28868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7fa366f73047f78897ec5cdfa75749" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260812 11:20:05.693334 28868 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b7fa366f73047f78897ec5cdfa75749; no voters:
I20260812 11:20:05.693791 28868 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:05.694064 28884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:05.694456 28884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [term 1 LEADER]: Becoming Leader. State: Replica: 9b7fa366f73047f78897ec5cdfa75749, State: Running, Role: LEADER
I20260812 11:20:05.695065 28884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7fa366f73047f78897ec5cdfa75749" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260812 11:20:05.695748 28868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:05.697836 28886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b7fa366f73047f78897ec5cdfa75749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7fa366f73047f78897ec5cdfa75749" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } } }
I20260812 11:20:05.698061 28886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:05.697815 28887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b7fa366f73047f78897ec5cdfa75749. Latest consensus state: current_term: 1 leader_uuid: "9b7fa366f73047f78897ec5cdfa75749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7fa366f73047f78897ec5cdfa75749" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } } }
I20260812 11:20:05.698169 28887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:05.698724 28904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:05.703267 28904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:05.711930 28904 catalog_manager.cc:1347] Generated new cluster ID: 66a8f8d5dffe450883a825d42c41cb09
I20260812 11:20:05.712062 28904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:05.723142 28904 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:05.723666 28904 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:05.737473 28904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b7fa366f73047f78897ec5cdfa75749: Generated new TSK 0
I20260812 11:20:05.738579 28904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:05.742707 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28757
2026-08-12T11:20:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:05 dist-test-slave-7z32 krb5kdc[28921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:05 dist-test-slave-7z32 krb5kdc[28921](info): set up 2 sockets
Aug 12 11:20:05 dist-test-slave-7z32 krb5kdc[28921](info): commencing operation
krb5kdc: starting...
W20260812 11:20:07.123587 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.338s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:07 dist-test-slave-7z32 krb5kdc[28921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:07.291512 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:07.447757 28948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:07.448225 28948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:07.448347 28948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:07.453457 28948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:07.453603 28948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:07.453655 28948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:07.453718 28948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:07.453763 28948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:07.461042 28948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:07.462721 28948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:07.464176 28948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:07.476054 28957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:07.475996 28958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:07.476006 28963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:07.476490 28948 server_base.cc:1034] running on GCE node
I20260812 11:20:07.477465 28948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:07.479496 28948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:07.480792 28948 hybrid_clock.cc:648] HybridClock initialized: now 1786533607480749 us; error 63 us; skew 500 ppm
Aug 12 11:20:07 dist-test-slave-7z32 krb5kdc[28921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:07.485162 28948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:07.486680 28948 webserver.cc:460] Webserver started at http://127.0.0.1:46081/ using document root <none> and password file <none>
I20260812 11:20:07.487556 28948 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:07.487680 28948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:07.488008 28948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:07.491071 28948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "3d0c2d40ccc94d9688febec1d15c297e"
format_stamp: "Formatted at 2026-08-12 11:20:07 on dist-test-slave-7z32"
server_key: "95fb37ce235202b8f32c282735b8a880"
server_key_iv: "80c65e9ed818b273debd80bd4c24c9d9"
server_key_version: "encryptionkey@0"
I20260812 11:20:07.491901 28948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "3d0c2d40ccc94d9688febec1d15c297e"
format_stamp: "Formatted at 2026-08-12 11:20:07 on dist-test-slave-7z32"
server_key: "95fb37ce235202b8f32c282735b8a880"
server_key_iv: "80c65e9ed818b273debd80bd4c24c9d9"
server_key_version: "encryptionkey@0"
I20260812 11:20:07.497611 28948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:20:07.501758 28973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:07.504031 28948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:20:07.504317 28948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "3d0c2d40ccc94d9688febec1d15c297e"
format_stamp: "Formatted at 2026-08-12 11:20:07 on dist-test-slave-7z32"
server_key: "95fb37ce235202b8f32c282735b8a880"
server_key_iv: "80c65e9ed818b273debd80bd4c24c9d9"
server_key_version: "encryptionkey@0"
I20260812 11:20:07.504518 28948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:07.518342 28948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:07.523288 28948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:07.523633 28948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:07.534332 28948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45663
I20260812 11:20:07.534368 29059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45663 every 8 connection(s)
I20260812 11:20:07.535897 28948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:07.539601 29060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:07.543442 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 28948
I20260812 11:20:07.543628 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:07.543977 28084 external_mini_cluster.cc:1428] Setting key bfd11de409782892d906020d1f9282aa
I20260812 11:20:07.546756 29060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e: Bootstrap starting.
I20260812 11:20:07.549767 29060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:07.550660 29060 log.cc:824] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:07.556105 29060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e: No bootstrap required, opened a new log
Aug 12 11:20:07 dist-test-slave-7z32 krb5kdc[28921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:20:07.559711 29060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0c2d40ccc94d9688febec1d15c297e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260812 11:20:07.560035 29060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:07.560129 29060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0c2d40ccc94d9688febec1d15c297e, State: Initialized, Role: FOLLOWER
I20260812 11:20:07.560626 29060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0c2d40ccc94d9688febec1d15c297e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260812 11:20:07.560793 29060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:07.560930 29060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:07.561136 29060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:07.562237 29060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0c2d40ccc94d9688febec1d15c297e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260812 11:20:07.562634 29060 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0c2d40ccc94d9688febec1d15c297e; no voters:
I20260812 11:20:07.563087 29060 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:07.563303 29069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:07.563669 29069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [term 1 LEADER]: Becoming Leader. State: Replica: 3d0c2d40ccc94d9688febec1d15c297e, State: Running, Role: LEADER
I20260812 11:20:07.564147 29069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0c2d40ccc94d9688febec1d15c297e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260812 11:20:07.564692 29060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:07.565517 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:07.546042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57589 (local address 127.0.0.1:45663)
0812 11:20:07.546753 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:07.546789 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:07.546993 (+ 204us) server_negotiation.cc:407] Connection header received
0812 11:20:07.548166 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:07.548196 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:07.548754 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:07.549286 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:07.550447 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:07.551825 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:07.552912 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:07.553730 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:07.553954 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:07.557681 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:07.557688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:07.557710 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:07.557760 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:07.561052 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:07.561718 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:07.561734 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:07.561743 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:07.561909 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:07.562244 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:07.562251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:07.562254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:07.563039 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:07.563325 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:07.563986 (+ 661us) server_negotiation.cc:299] Negotiation successful
0812 11:20:07.564407 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":206,"threads_started":1}
I20260812 11:20:07.566679 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0c2d40ccc94d9688febec1d15c297e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0c2d40ccc94d9688febec1d15c297e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } } }
I20260812 11:20:07.566906 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:07.566890 29071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0c2d40ccc94d9688febec1d15c297e. Latest consensus state: current_term: 1 leader_uuid: "3d0c2d40ccc94d9688febec1d15c297e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0c2d40ccc94d9688febec1d15c297e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } } }
I20260812 11:20:07.567062 29071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:07.567515 29095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:07.570539 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:07.571955 29095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:07.581558 29095 catalog_manager.cc:1347] Generated new cluster ID: ce1ba40c8050473790ef16ac7c818c7b
I20260812 11:20:07.581648 29095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:07.585304 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:07.573315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50237 (local address 127.0.0.1:45663)
0812 11:20:07.573472 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:07.573477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:07.573854 (+ 377us) server_negotiation.cc:407] Connection header received
0812 11:20:07.574025 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:07.574029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:07.574094 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:07.574215 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:07.575349 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:07.576189 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:07.577616 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:07.578020 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:07.578156 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:07.580863 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:07.580867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:07.580870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:07.580906 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:07.583161 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:07.583735 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:07.583739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:07.583741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:07.583852 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:07.584160 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:07.584164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:07.584166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:07.584418 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:07.584575 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:07.584887 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 11:20:07.585035 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:20:07.595232 29095 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:07.596580 29095 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:07.607075 29095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0c2d40ccc94d9688febec1d15c297e: Generated new TSK 0
I20260812 11:20:07.608364 29095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:07.616197 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 28948
2026-08-12T11:20:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:07 dist-test-slave-7z32 krb5kdc[29110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:07 dist-test-slave-7z32 krb5kdc[29110](info): set up 2 sockets
Aug 12 11:20:07 dist-test-slave-7z32 krb5kdc[29110](info): commencing operation
krb5kdc: starting...
W20260812 11:20:08.978178 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:09 dist-test-slave-7z32 krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:09.147392 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:09.309113 29130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:09.309497 29130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:09.309546 29130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:09.315397 29130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:09.315539 29130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:09.315563 29130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:09.315583 29130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:09.315599 29130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:09.324774 29130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:09.326361 29130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:09.327898 29130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:09.342490 29136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:09.342491 29139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:09.342769 29137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:09.342804 29130 server_base.cc:1034] running on GCE node
I20260812 11:20:09.344146 29130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:09.345849 29130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:09.349298 29130 hybrid_clock.cc:648] HybridClock initialized: now 1786533609349158 us; error 149 us; skew 500 ppm
Aug 12 11:20:09 dist-test-slave-7z32 krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:09.360597 29130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:09.361986 29130 webserver.cc:460] Webserver started at http://127.0.0.1:42801/ using document root <none> and password file <none>
I20260812 11:20:09.363014 29130 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:09.363152 29130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:09.363651 29130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:09.367085 29130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "83b7becc5fb24485ae64d112d7f1f60a"
format_stamp: "Formatted at 2026-08-12 11:20:09 on dist-test-slave-7z32"
server_key: "cf93a79c8754022314ed133dd2495079"
server_key_iv: "8db87387b382b866edfbb7d0ede0a6be"
server_key_version: "encryptionkey@0"
I20260812 11:20:09.368026 29130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "83b7becc5fb24485ae64d112d7f1f60a"
format_stamp: "Formatted at 2026-08-12 11:20:09 on dist-test-slave-7z32"
server_key: "cf93a79c8754022314ed133dd2495079"
server_key_iv: "8db87387b382b866edfbb7d0ede0a6be"
server_key_version: "encryptionkey@0"
I20260812 11:20:09.373832 29130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:20:09.378129 29154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:09.380957 29130 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:20:09.381327 29130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "83b7becc5fb24485ae64d112d7f1f60a"
format_stamp: "Formatted at 2026-08-12 11:20:09 on dist-test-slave-7z32"
server_key: "cf93a79c8754022314ed133dd2495079"
server_key_iv: "8db87387b382b866edfbb7d0ede0a6be"
server_key_version: "encryptionkey@0"
I20260812 11:20:09.381525 29130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:09.408571 29130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:09.413667 29130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:09.413972 29130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:09.425998 29130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46815
I20260812 11:20:09.426111 29233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46815 every 8 connection(s)
I20260812 11:20:09.427471 29130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:09.432487 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29130
I20260812 11:20:09.432706 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:09.433022 29235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:09.433120 28084 external_mini_cluster.cc:1428] Setting key e5b98db6ad7e28093ec73917f8637a53
I20260812 11:20:09.443270 29235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a: Bootstrap starting.
I20260812 11:20:09.446044 29241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:09.435352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39351 (local address 127.0.0.1:46815)
0812 11:20:09.436190 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:09.436212 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:09.436271 (+ 59us) server_negotiation.cc:407] Connection header received
0812 11:20:09.437311 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:09.437337 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:09.437834 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:09.438319 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:09.438956 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:09.440221 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:09.441379 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:09.442030 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:09.442261 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:09.443569 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:09.443576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:09.443595 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:09.444242 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:09.444373 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:09.444407 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:09.444563 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:09.444765 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 11:20:09.445063 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":255,"threads_started":1}
I20260812 11:20:09.447134 29235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:09.448391 29235 log.cc:824] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:09.451004 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:09.452435 29235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a: No bootstrap required, opened a new log
I20260812 11:20:09.457948 29235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b7becc5fb24485ae64d112d7f1f60a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260812 11:20:09.458536 29235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:09.458600 29235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b7becc5fb24485ae64d112d7f1f60a, State: Initialized, Role: FOLLOWER
I20260812 11:20:09.459782 29235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b7becc5fb24485ae64d112d7f1f60a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260812 11:20:09.460089 29235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:09.460280 29235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:09.460455 29235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:09.461436 29241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:09.454458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58241 (local address 127.0.0.1:46815)
0812 11:20:09.454671 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:09.454677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:09.455056 (+ 379us) server_negotiation.cc:407] Connection header received
0812 11:20:09.455304 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:09.455309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:09.455389 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:09.455500 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:09.455998 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:09.457076 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:09.458614 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:09.459121 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:09.459251 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:09.460926 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:09.460929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:09.460931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:09.460949 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:09.460983 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:09.460995 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:09.461060 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:09.461205 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 11:20:09.461268 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:20:09.462288 29235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b7becc5fb24485ae64d112d7f1f60a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260812 11:20:09.463068 29235 leader_election.cc:302] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b7becc5fb24485ae64d112d7f1f60a; no voters:
I20260812 11:20:09.463605 29235 leader_election.cc:288] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:09.463757 29254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:09.464131 29254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [term 1 LEADER]: Becoming Leader. State: Replica: 83b7becc5fb24485ae64d112d7f1f60a, State: Running, Role: LEADER
I20260812 11:20:09.464597 29254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b7becc5fb24485ae64d112d7f1f60a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260812 11:20:09.465451 29235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:09.466756 29255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83b7becc5fb24485ae64d112d7f1f60a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b7becc5fb24485ae64d112d7f1f60a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260812 11:20:09.466756 29256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b7becc5fb24485ae64d112d7f1f60a. Latest consensus state: current_term: 1 leader_uuid: "83b7becc5fb24485ae64d112d7f1f60a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b7becc5fb24485ae64d112d7f1f60a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260812 11:20:09.467020 29256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:09.467374 29255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:09.467545 29264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:09.472289 29264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:09.484934 29264 catalog_manager.cc:1347] Generated new cluster ID: b3d9fb54ad8e46e98b2c90a6b6a27f26
I20260812 11:20:09.485102 29264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:09.499243 29264 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:09.500867 29264 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:09.514178 29264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83b7becc5fb24485ae64d112d7f1f60a: Generated new TSK 0
I20260812 11:20:09.516040 29264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:09.531219 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29130
2026-08-12T11:20:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:09 dist-test-slave-7z32 krb5kdc[29279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:09 dist-test-slave-7z32 krb5kdc[29279](info): set up 2 sockets
Aug 12 11:20:09 dist-test-slave-7z32 krb5kdc[29279](info): commencing operation
krb5kdc: starting...
W20260812 11:20:10.969388 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.384s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:11 dist-test-slave-7z32 krb5kdc[29279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:11.171242 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:11.341456 29299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:11.342017 29299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:11.342077 29299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:11.347499 29299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:11.347627 29299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:11.347651 29299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:11.347678 29299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:11.347697 29299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:11.355060 29299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:11.356750 29299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:11.358230 29299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:20:11.370481 29299 server_base.cc:1034] running on GCE node
W20260812 11:20:11.370419 29309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:11.370419 29305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:11.370431 29306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:11.371615 29299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:11.372996 29299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:11.374229 29299 hybrid_clock.cc:648] HybridClock initialized: now 1786533611374210 us; error 48 us; skew 500 ppm
Aug 12 11:20:11 dist-test-slave-7z32 krb5kdc[29279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:11.378114 29299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:11.379477 29299 webserver.cc:460] Webserver started at http://127.0.0.1:37197/ using document root <none> and password file <none>
I20260812 11:20:11.380206 29299 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:11.380293 29299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:11.380566 29299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:11.383183 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "d355d3368d5d424c8c9e04033ac5b226"
format_stamp: "Formatted at 2026-08-12 11:20:11 on dist-test-slave-7z32"
server_key: "6153847269cd325d1cb566f319e8157c"
server_key_iv: "b0e2879cb6dc8a20fc94611c991db042"
server_key_version: "encryptionkey@0"
I20260812 11:20:11.383949 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "d355d3368d5d424c8c9e04033ac5b226"
format_stamp: "Formatted at 2026-08-12 11:20:11 on dist-test-slave-7z32"
server_key: "6153847269cd325d1cb566f319e8157c"
server_key_iv: "b0e2879cb6dc8a20fc94611c991db042"
server_key_version: "encryptionkey@0"
I20260812 11:20:11.389014 29299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:20:11.392868 29319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:11.395174 29299 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:20:11.395428 29299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "d355d3368d5d424c8c9e04033ac5b226"
format_stamp: "Formatted at 2026-08-12 11:20:11 on dist-test-slave-7z32"
server_key: "6153847269cd325d1cb566f319e8157c"
server_key_iv: "b0e2879cb6dc8a20fc94611c991db042"
server_key_version: "encryptionkey@0"
I20260812 11:20:11.395608 29299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:11.401697 29299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:11.406430 29299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:11.406698 29299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:11.416167 29299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34609
I20260812 11:20:11.416206 29399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34609 every 8 connection(s)
I20260812 11:20:11.417419 29299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:11.422329 29400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:11.424924 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29299
I20260812 11:20:11.425143 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:11.425496 28084 external_mini_cluster.cc:1428] Setting key 4b79ae5843e71877369f4cd933c23f56
I20260812 11:20:11.432250 29400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226: Bootstrap starting.
I20260812 11:20:11.432778 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:11.427607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34785 (local address 127.0.0.1:34609)
0812 11:20:11.428199 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:11.428216 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:11.428260 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:20:11.429123 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:11.429145 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:11.429485 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:11.429835 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:11.431146 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:11.431153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:11.431167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:11.431181 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:11.431251 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:11.431274 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:11.431462 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:11.431583 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 11:20:11.431786 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":207,"threads_started":1}
I20260812 11:20:11.435969 29400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:11.437266 29400 log.cc:824] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:11.437865 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:11.440517 29400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226: No bootstrap required, opened a new log
I20260812 11:20:11.443697 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:11.440921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54115 (local address 127.0.0.1:34609)
0812 11:20:11.441096 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:11.441101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:11.441445 (+ 344us) server_negotiation.cc:407] Connection header received
0812 11:20:11.441515 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:11.441517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:11.441564 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:11.441658 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:11.442695 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:11.442699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:11.442701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:11.442717 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:11.442755 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:11.443084 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:11.443198 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:11.443420 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 11:20:11.443509 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:20:11.446645 29400 raft_consensus.cc:348] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d355d3368d5d424c8c9e04033ac5b226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260812 11:20:11.447155 29400 raft_consensus.cc:374] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:11.447256 29400 raft_consensus.cc:729] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d355d3368d5d424c8c9e04033ac5b226, State: Initialized, Role: FOLLOWER
I20260812 11:20:11.447862 29400 consensus_queue.cc:260] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d355d3368d5d424c8c9e04033ac5b226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260812 11:20:11.448052 29400 raft_consensus.cc:388] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:11.448146 29400 raft_consensus.cc:482] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:11.448297 29400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:11.449707 29400 raft_consensus.cc:504] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d355d3368d5d424c8c9e04033ac5b226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260812 11:20:11.450268 29400 leader_election.cc:302] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d355d3368d5d424c8c9e04033ac5b226; no voters:
I20260812 11:20:11.451033 29400 leader_election.cc:288] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:11.451251 29413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:11.451660 29413 raft_consensus.cc:686] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [term 1 LEADER]: Becoming Leader. State: Replica: d355d3368d5d424c8c9e04033ac5b226, State: Running, Role: LEADER
I20260812 11:20:11.452145 29413 consensus_queue.cc:237] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d355d3368d5d424c8c9e04033ac5b226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260812 11:20:11.453518 29400 sys_catalog.cc:564] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:11.454809 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d355d3368d5d424c8c9e04033ac5b226. Latest consensus state: current_term: 1 leader_uuid: "d355d3368d5d424c8c9e04033ac5b226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d355d3368d5d424c8c9e04033ac5b226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } } }
I20260812 11:20:11.454882 29414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d355d3368d5d424c8c9e04033ac5b226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d355d3368d5d424c8c9e04033ac5b226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } } }
I20260812 11:20:11.455071 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:11.455071 29414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:11.455756 29427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:11.460242 29427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:11.469522 29427 catalog_manager.cc:1347] Generated new cluster ID: d1c8793b11b34d48823b10b52a1e2ce4
I20260812 11:20:11.469633 29427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:11.486253 29427 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:11.486949 29427 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:11.495198 29427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d355d3368d5d424c8c9e04033ac5b226: Generated new TSK 0
I20260812 11:20:11.496245 29427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:11.509083 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29299
2026-08-12T11:20:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:11 dist-test-slave-7z32 krb5kdc[29445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:11 dist-test-slave-7z32 krb5kdc[29445](info): set up 2 sockets
Aug 12 11:20:11 dist-test-slave-7z32 krb5kdc[29445](info): commencing operation
krb5kdc: starting...
W20260812 11:20:12.959326 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.405s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:13 dist-test-slave-7z32 krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:13.128489 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:13.260329 29466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:13.260668 29466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:13.260713 29466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:13.265228 29466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:13.265329 29466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:13.265350 29466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:13.265367 29466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:13.265384 29466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:13.271347 29466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:13.272681 29466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:13.273836 29466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:13.285270 29477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:13.285270 29474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:13.285317 29473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:13.285688 29466 server_base.cc:1034] running on GCE node
I20260812 11:20:13.286648 29466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:13.288560 29466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:13.289798 29466 hybrid_clock.cc:648] HybridClock initialized: now 1786533613289754 us; error 63 us; skew 500 ppm
Aug 12 11:20:13 dist-test-slave-7z32 krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:13.293905 29466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:13.295156 29466 webserver.cc:460] Webserver started at http://127.0.0.1:39695/ using document root <none> and password file <none>
I20260812 11:20:13.295796 29466 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:13.295871 29466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:13.296064 29466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:13.298457 29466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "20a983956b004503b171dcdc91b61263"
format_stamp: "Formatted at 2026-08-12 11:20:13 on dist-test-slave-7z32"
server_key: "21e3d5728301b82da3d78f034634d183"
server_key_iv: "1a3a6717fbca7fe6fca92fef306efabb"
server_key_version: "encryptionkey@0"
I20260812 11:20:13.299147 29466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "20a983956b004503b171dcdc91b61263"
format_stamp: "Formatted at 2026-08-12 11:20:13 on dist-test-slave-7z32"
server_key: "21e3d5728301b82da3d78f034634d183"
server_key_iv: "1a3a6717fbca7fe6fca92fef306efabb"
server_key_version: "encryptionkey@0"
I20260812 11:20:13.304035 29466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:20:13.307637 29487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:13.309451 29466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:20:13.309619 29466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "20a983956b004503b171dcdc91b61263"
format_stamp: "Formatted at 2026-08-12 11:20:13 on dist-test-slave-7z32"
server_key: "21e3d5728301b82da3d78f034634d183"
server_key_iv: "1a3a6717fbca7fe6fca92fef306efabb"
server_key_version: "encryptionkey@0"
I20260812 11:20:13.309751 29466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:13.332173 29466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:13.336318 29466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:13.336519 29466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:13.345714 29466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40783
I20260812 11:20:13.345767 29577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40783 every 8 connection(s)
I20260812 11:20:13.347052 29466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:13.349315 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29466
I20260812 11:20:13.349495 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:13.349815 28084 external_mini_cluster.cc:1428] Setting key 0bc9ff58a92b920789fda5296c1efba9
I20260812 11:20:13.352488 29578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 11:20:13 dist-test-slave-7z32 krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:20:13.361181 29578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263: Bootstrap starting.
I20260812 11:20:13.365028 29578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:13.366147 29578 log.cc:824] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:13.367887 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:13.351899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:40423 (local address 127.0.0.1:40783)
0812 11:20:13.352762 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:13.352784 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:13.352842 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:20:13.353729 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:13.353754 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:13.354192 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:13.354663 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:13.355592 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:13.356675 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:13.357610 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:13.358129 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:13.358344 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:13.361513 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:13.361519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:13.361537 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:13.361575 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:13.363871 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:13.364564 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:13.364570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:13.364577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:13.364781 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:13.365194 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:13.365198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:13.365202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:13.365748 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:13.366004 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:13.366409 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 11:20:13.366686 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":233,"threads_started":1}
I20260812 11:20:13.369927 29578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263: No bootstrap required, opened a new log
I20260812 11:20:13.372695 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:13.375857 29578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a983956b004503b171dcdc91b61263" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40783 } }
I20260812 11:20:13.376314 29578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:13.376433 29578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a983956b004503b171dcdc91b61263, State: Initialized, Role: FOLLOWER
I20260812 11:20:13.377174 29578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a983956b004503b171dcdc91b61263" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40783 } }
I20260812 11:20:13.377381 29578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:13.377467 29578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:13.377591 29578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:13.379226 29578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a983956b004503b171dcdc91b61263" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40783 } }
I20260812 11:20:13.379739 29578 leader_election.cc:302] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20a983956b004503b171dcdc91b61263; no voters:
I20260812 11:20:13.380311 29578 leader_election.cc:288] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:13.380813 29595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:13.381074 29595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [term 1 LEADER]: Becoming Leader. State: Replica: 20a983956b004503b171dcdc91b61263, State: Running, Role: LEADER
I20260812 11:20:13.382085 29595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a983956b004503b171dcdc91b61263" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40783 } }
I20260812 11:20:13.382376 29578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:13.384866 29596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20a983956b004503b171dcdc91b61263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a983956b004503b171dcdc91b61263" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40783 } } }
I20260812 11:20:13.385164 29596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:13.385643 29594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20a983956b004503b171dcdc91b61263. Latest consensus state: current_term: 1 leader_uuid: "20a983956b004503b171dcdc91b61263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a983956b004503b171dcdc91b61263" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40783 } } }
I20260812 11:20:13.386300 29594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:13.385751 29612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:13.386626 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:13.375278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:54905 (local address 127.0.0.1:40783)
0812 11:20:13.375445 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:13.375451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:13.375584 (+ 133us) server_negotiation.cc:407] Connection header received
0812 11:20:13.375682 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:13.375686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:13.375757 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:13.375883 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:13.376796 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:13.377724 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:13.378665 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:13.379160 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:13.379309 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:13.381538 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:13.381542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:13.381546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:13.381596 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:13.383827 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:13.384629 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:13.384633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:13.384635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:13.384753 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:13.385216 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:13.385219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:13.385221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:13.385524 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:13.385688 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:13.386125 (+ 437us) server_negotiation.cc:299] Negotiation successful
0812 11:20:13.386239 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:20:13.391070 29612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:13.400306 29612 catalog_manager.cc:1347] Generated new cluster ID: 917151fef0214179a2240a8f2f02df83
I20260812 11:20:13.400456 29612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:13.410560 29612 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:13.412081 29612 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:13.417275 29612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20a983956b004503b171dcdc91b61263: Generated new TSK 0
I20260812 11:20:13.418359 29612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:13.423627 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29466
W20260812 11:20:13.456409 29443 connection.cc:535] client connection to 127.0.0.1:40783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40783): Connection reset by peer (error 104)
2026-08-12T11:20:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:13 dist-test-slave-7z32 krb5kdc[29624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:13 dist-test-slave-7z32 krb5kdc[29624](info): set up 2 sockets
Aug 12 11:20:13 dist-test-slave-7z32 krb5kdc[29624](info): commencing operation
krb5kdc: starting...
W20260812 11:20:14.810932 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:14 dist-test-slave-7z32 krb5kdc[29624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:14.979637 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:15.117661 29648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:15.118005 29648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:15.118052 29648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:15.122980 29648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:15.123090 29648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:15.123114 29648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:15.123138 29648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:15.123157 29648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:15.129640 29648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:15.131018 29648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:15.132241 29648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:15.143581 29662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:15.143622 29656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:15.143622 29655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:15.144315 29648 server_base.cc:1034] running on GCE node
I20260812 11:20:15.145201 29648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:15.146924 29648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:15.148175 29648 hybrid_clock.cc:648] HybridClock initialized: now 1786533615148054 us; error 141 us; skew 500 ppm
Aug 12 11:20:15 dist-test-slave-7z32 krb5kdc[29624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:15.157490 29648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:15.158782 29648 webserver.cc:460] Webserver started at http://127.0.0.1:41237/ using document root <none> and password file <none>
I20260812 11:20:15.159685 29648 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:15.159782 29648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:15.160073 29648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:15.162626 29648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "0203795acfa540b0990cc3e23e88de19"
format_stamp: "Formatted at 2026-08-12 11:20:15 on dist-test-slave-7z32"
server_key: "e9ed010ba102c722e17a005921395991"
server_key_iv: "29f08a9676e9cbcc9c20063aff0379f5"
server_key_version: "encryptionkey@0"
I20260812 11:20:15.163407 29648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "0203795acfa540b0990cc3e23e88de19"
format_stamp: "Formatted at 2026-08-12 11:20:15 on dist-test-slave-7z32"
server_key: "e9ed010ba102c722e17a005921395991"
server_key_iv: "29f08a9676e9cbcc9c20063aff0379f5"
server_key_version: "encryptionkey@0"
I20260812 11:20:15.169308 29648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:20:15.173530 29673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:15.175583 29648 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:20:15.175823 29648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0203795acfa540b0990cc3e23e88de19"
format_stamp: "Formatted at 2026-08-12 11:20:15 on dist-test-slave-7z32"
server_key: "e9ed010ba102c722e17a005921395991"
server_key_iv: "29f08a9676e9cbcc9c20063aff0379f5"
server_key_version: "encryptionkey@0"
I20260812 11:20:15.175985 29648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:15.202041 29648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:15.206449 29648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:15.206705 29648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:15.215862 29648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34475
I20260812 11:20:15.216001 29754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34475 every 8 connection(s)
I20260812 11:20:15.217082 29648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:15.221302 29761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:15.222386 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29648
I20260812 11:20:15.222579 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:15.223031 28084 external_mini_cluster.cc:1428] Setting key c3c72b218b28ed08cb502a730b1373bb
I20260812 11:20:15.231096 29761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19: Bootstrap starting.
I20260812 11:20:15.234790 29761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:15.235181 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:15.225093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:53277 (local address 127.0.0.1:34475)
0812 11:20:15.225908 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:15.225939 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:15.225997 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:20:15.227078 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:15.227106 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:15.227610 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:15.228052 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:15.228556 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:15.229706 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:15.230774 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:15.231548 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:15.231755 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:15.232986 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:15.232991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:15.233011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:15.233490 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:15.233561 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:15.233587 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:15.233746 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:15.233961 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 11:20:15.234141 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":234,"threads_started":1}
I20260812 11:20:15.236038 29761 log.cc:824] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:15.239384 29761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19: No bootstrap required, opened a new log
I20260812 11:20:15.239670 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:15.245633 29761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0203795acfa540b0990cc3e23e88de19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260812 11:20:15.246348 29761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:15.246475 29761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0203795acfa540b0990cc3e23e88de19, State: Initialized, Role: FOLLOWER
I20260812 11:20:15.247558 29761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0203795acfa540b0990cc3e23e88de19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260812 11:20:15.247886 29761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:15.247992 29761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:15.248158 29761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:15.248592 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:15.242217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:36285 (local address 127.0.0.1:34475)
0812 11:20:15.242375 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:15.242379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:15.242593 (+ 214us) server_negotiation.cc:407] Connection header received
0812 11:20:15.242801 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:15.242803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:15.242912 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:15.243002 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:15.243555 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:15.244135 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:15.245528 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:15.245836 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:15.246069 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:15.247840 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:15.247846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:15.247852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:15.247874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:15.247934 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:15.247970 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:15.248097 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:15.248350 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 11:20:15.248415 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:20:15.250159 29761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0203795acfa540b0990cc3e23e88de19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260812 11:20:15.250756 29761 leader_election.cc:302] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0203795acfa540b0990cc3e23e88de19; no voters:
I20260812 11:20:15.251374 29761 leader_election.cc:288] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:15.251582 29776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:15.251965 29776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [term 1 LEADER]: Becoming Leader. State: Replica: 0203795acfa540b0990cc3e23e88de19, State: Running, Role: LEADER
I20260812 11:20:15.252497 29776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0203795acfa540b0990cc3e23e88de19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260812 11:20:15.253326 29761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:15.254799 29779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0203795acfa540b0990cc3e23e88de19. Latest consensus state: current_term: 1 leader_uuid: "0203795acfa540b0990cc3e23e88de19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0203795acfa540b0990cc3e23e88de19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } } }
I20260812 11:20:15.255107 29779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:15.255554 29778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0203795acfa540b0990cc3e23e88de19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0203795acfa540b0990cc3e23e88de19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } } }
I20260812 11:20:15.255789 29778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:15.256422 29794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:15.260156 29794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:15.266628 29794 catalog_manager.cc:1347] Generated new cluster ID: fc46d7b21c6e4bf4bf3f15a8ad63c7e7
I20260812 11:20:15.266737 29794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:15.274480 29794 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:15.275810 29794 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:15.288619 29794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0203795acfa540b0990cc3e23e88de19: Generated new TSK 0
I20260812 11:20:15.289579 29794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:15.301651 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29648
W20260812 11:20:15.313737 29619 connection.cc:535] client connection to 127.0.0.1:34475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34475): Connection reset by peer (error 104)
2026-08-12T11:20:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:15 dist-test-slave-7z32 krb5kdc[29813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:15 dist-test-slave-7z32 krb5kdc[29813](info): set up 2 sockets
Aug 12 11:20:15 dist-test-slave-7z32 krb5kdc[29813](info): commencing operation
krb5kdc: starting...
W20260812 11:20:16.705379 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.376s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:16 dist-test-slave-7z32 krb5kdc[29813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:16.876471 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:17.038439 29834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:17.039004 29834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:17.039139 29834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:17.044911 29834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:17.045058 29834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:17.045109 29834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:17.045166 29834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:17.045204 29834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:17.052918 29834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:17.054797 29834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:17.057210 29834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:17.073048 29842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:17.073117 29841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:17.073228 29834 server_base.cc:1034] running on GCE node
W20260812 11:20:17.073073 29844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:17.074698 29834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:17.076283 29834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:17.079638 29834 hybrid_clock.cc:648] HybridClock initialized: now 1786533617079536 us; error 172 us; skew 500 ppm
Aug 12 11:20:17 dist-test-slave-7z32 krb5kdc[29813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:17.083582 29834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:17.085140 29834 webserver.cc:460] Webserver started at http://127.0.0.1:36849/ using document root <none> and password file <none>
I20260812 11:20:17.085960 29834 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:17.086047 29834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:17.086333 29834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:17.089748 29834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "40fd5a4db4034b299a77b495f50554d7"
format_stamp: "Formatted at 2026-08-12 11:20:17 on dist-test-slave-7z32"
server_key: "f795799e529cf439fd2c122683b6d95c"
server_key_iv: "00ddb4c3a4dae744692abaa6c7a87e0a"
server_key_version: "encryptionkey@0"
I20260812 11:20:17.090727 29834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "40fd5a4db4034b299a77b495f50554d7"
format_stamp: "Formatted at 2026-08-12 11:20:17 on dist-test-slave-7z32"
server_key: "f795799e529cf439fd2c122683b6d95c"
server_key_iv: "00ddb4c3a4dae744692abaa6c7a87e0a"
server_key_version: "encryptionkey@0"
I20260812 11:20:17.097730 29834 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 11:20:17.102373 29853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:17.105083 29834 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:20:17.105386 29834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "40fd5a4db4034b299a77b495f50554d7"
format_stamp: "Formatted at 2026-08-12 11:20:17 on dist-test-slave-7z32"
server_key: "f795799e529cf439fd2c122683b6d95c"
server_key_iv: "00ddb4c3a4dae744692abaa6c7a87e0a"
server_key_version: "encryptionkey@0"
I20260812 11:20:17.105602 29834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:17.113078 29834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:17.118148 29834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:17.118497 29834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:17.129001 29834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35457
I20260812 11:20:17.129019 29938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35457 every 8 connection(s)
I20260812 11:20:17.130467 29834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:17.134402 29941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:17.138850 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 29834
I20260812 11:20:17.139021 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:17.139350 28084 external_mini_cluster.cc:1428] Setting key ddbf53b478b6de13d706380ca99cf376
I20260812 11:20:17.142161 29941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7: Bootstrap starting.
I20260812 11:20:17.145232 29941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:17.146272 29941 log.cc:824] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:17.148687 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:17.141462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:40841 (local address 127.0.0.1:35457)
0812 11:20:17.142137 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:17.142170 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:17.142222 (+ 52us) server_negotiation.cc:407] Connection header received
0812 11:20:17.143294 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:17.143322 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:17.143756 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:17.144407 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:17.145817 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:17.145828 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:17.145852 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:17.146332 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:17.146509 (+ 177us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:17.146552 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:17.146733 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:17.147034 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 11:20:17.147317 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":226,"threads_started":1}
I20260812 11:20:17.149672 29941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7: No bootstrap required, opened a new log
I20260812 11:20:17.154093 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:17.155961 29941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fd5a4db4034b299a77b495f50554d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260812 11:20:17.156500 29941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:17.156594 29941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fd5a4db4034b299a77b495f50554d7, State: Initialized, Role: FOLLOWER
I20260812 11:20:17.157167 29941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fd5a4db4034b299a77b495f50554d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260812 11:20:17.157346 29941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:17.157433 29941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:17.157563 29941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:17.159253 29941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fd5a4db4034b299a77b495f50554d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260812 11:20:17.159754 29941 leader_election.cc:302] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40fd5a4db4034b299a77b495f50554d7; no voters:
I20260812 11:20:17.160264 29941 leader_election.cc:288] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:17.160437 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:17.157717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:52255 (local address 127.0.0.1:35457)
0812 11:20:17.157978 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:17.157983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:17.158135 (+ 152us) server_negotiation.cc:407] Connection header received
0812 11:20:17.158330 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:17.158333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:17.158387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:17.158485 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:17.159542 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:17.159547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:17.159551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:17.159567 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:17.159616 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:17.159630 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:17.159793 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:17.160021 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 11:20:17.160167 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260812 11:20:17.160481 29953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:17.160883 29953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [term 1 LEADER]: Becoming Leader. State: Replica: 40fd5a4db4034b299a77b495f50554d7, State: Running, Role: LEADER
I20260812 11:20:17.161479 29953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fd5a4db4034b299a77b495f50554d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260812 11:20:17.161983 29941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:17.163437 29953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40fd5a4db4034b299a77b495f50554d7. Latest consensus state: current_term: 1 leader_uuid: "40fd5a4db4034b299a77b495f50554d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fd5a4db4034b299a77b495f50554d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } } }
I20260812 11:20:17.163754 29953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:17.164342 29956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40fd5a4db4034b299a77b495f50554d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fd5a4db4034b299a77b495f50554d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } } }
I20260812 11:20:17.164589 29956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7 [sys.catalog]: This master's current role is: LEADER
W20260812 11:20:17.168979 29978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:17.169121 29978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:17.169293 29971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:17.170599 29971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:17.179323 29971 catalog_manager.cc:1347] Generated new cluster ID: bd88a66d435a4983b38a1c235b0efa2e
I20260812 11:20:17.179474 29971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:17.207605 29971 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:17.208213 29971 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:17.215562 29971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40fd5a4db4034b299a77b495f50554d7: Generated new TSK 0
I20260812 11:20:17.216714 29971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:17.221441 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 29834
2026-08-12T11:20:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1922 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:17 dist-test-slave-7z32 krb5kdc[29986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:17 dist-test-slave-7z32 krb5kdc[29986](info): set up 2 sockets
Aug 12 11:20:17 dist-test-slave-7z32 krb5kdc[29986](info): commencing operation
krb5kdc: starting...
W20260812 11:20:18.622771 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.370s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:18 dist-test-slave-7z32 krb5kdc[29986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:18.801146 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:18.935163 30004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:18.935563 30004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:18.935649 30004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:18.940035 30004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:18.940176 30004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:18.940215 30004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:18.940253 30004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:18.940290 30004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:18.946449 30004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:18.948280 30004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:18.949730 30004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:18.961061 30015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:18.961061 30012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:18.961337 30010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:18.962139 30004 server_base.cc:1034] running on GCE node
I20260812 11:20:18.963092 30004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:18.964555 30004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:18.965965 30004 hybrid_clock.cc:648] HybridClock initialized: now 1786533618965928 us; error 57 us; skew 500 ppm
Aug 12 11:20:18 dist-test-slave-7z32 krb5kdc[29986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:18.970057 30004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:18.971455 30004 webserver.cc:460] Webserver started at http://127.0.0.1:36567/ using document root <none> and password file <none>
I20260812 11:20:18.972137 30004 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:18.972244 30004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:18.972522 30004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:18.975274 30004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "9cefe84bdcb846968ff259a56ac6e08d"
format_stamp: "Formatted at 2026-08-12 11:20:18 on dist-test-slave-7z32"
server_key: "9854c60f94436b46feb1091d90a69ad5"
server_key_iv: "5ab435ed757e7530628446fad882df53"
server_key_version: "encryptionkey@0"
I20260812 11:20:18.976025 30004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "9cefe84bdcb846968ff259a56ac6e08d"
format_stamp: "Formatted at 2026-08-12 11:20:18 on dist-test-slave-7z32"
server_key: "9854c60f94436b46feb1091d90a69ad5"
server_key_iv: "5ab435ed757e7530628446fad882df53"
server_key_version: "encryptionkey@0"
I20260812 11:20:18.981827 30004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:20:18.985605 30028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:18.987564 30004 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:20:18.987780 30004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "9cefe84bdcb846968ff259a56ac6e08d"
format_stamp: "Formatted at 2026-08-12 11:20:18 on dist-test-slave-7z32"
server_key: "9854c60f94436b46feb1091d90a69ad5"
server_key_iv: "5ab435ed757e7530628446fad882df53"
server_key_version: "encryptionkey@0"
I20260812 11:20:18.987943 30004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:19.005681 30004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:19.009984 30004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:19.010257 30004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:19.020067 30004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39941
I20260812 11:20:19.020150 30100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39941 every 8 connection(s)
I20260812 11:20:19.021332 30004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:19.023952 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30004
I20260812 11:20:19.024345 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:19.025002 28084 external_mini_cluster.cc:1428] Setting key b27eec25be69416cd49b2337ba8cb0ff
I20260812 11:20:19.027107 30103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:19.035950 30103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d: Bootstrap starting.
Aug 12 11:20:19 dist-test-slave-7z32 krb5kdc[29986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:20:19.039659 30103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:19.040848 30103 log.cc:824] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:19.043995 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:19.027120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:47277 (local address 127.0.0.1:39941)
0812 11:20:19.027818 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:19.027839 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:19.027897 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:20:19.028944 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:19.028969 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:19.029411 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:19.029846 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:19.030806 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:19.032003 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:19.032940 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:19.033522 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:19.033717 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:19.037321 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:19.037328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:19.037348 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:19.037386 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:19.039663 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:19.040638 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:19.040645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:19.040651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:19.040800 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:19.041259 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:19.041262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:19.041263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:19.041965 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:19.042203 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:19.042676 (+ 473us) server_negotiation.cc:299] Negotiation successful
0812 11:20:19.043014 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":221,"threads_started":1}
I20260812 11:20:19.044531 30103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d: No bootstrap required, opened a new log
I20260812 11:20:19.047837 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:19.050328 30103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cefe84bdcb846968ff259a56ac6e08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260812 11:20:19.050783 30103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:19.050904 30103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cefe84bdcb846968ff259a56ac6e08d, State: Initialized, Role: FOLLOWER
I20260812 11:20:19.051728 30103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cefe84bdcb846968ff259a56ac6e08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260812 11:20:19.051997 30103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:19.052084 30103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:19.052227 30103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:19.054378 30103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cefe84bdcb846968ff259a56ac6e08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260812 11:20:19.055122 30103 leader_election.cc:302] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cefe84bdcb846968ff259a56ac6e08d; no voters:
I20260812 11:20:19.055735 30103 leader_election.cc:288] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:19.056293 30121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:19.056789 30121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [term 1 LEADER]: Becoming Leader. State: Replica: 9cefe84bdcb846968ff259a56ac6e08d, State: Running, Role: LEADER
I20260812 11:20:19.057500 30121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cefe84bdcb846968ff259a56ac6e08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260812 11:20:19.058606 30103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:19.059274 30118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cefe84bdcb846968ff259a56ac6e08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cefe84bdcb846968ff259a56ac6e08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } } }
I20260812 11:20:19.059360 30122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cefe84bdcb846968ff259a56ac6e08d. Latest consensus state: current_term: 1 leader_uuid: "9cefe84bdcb846968ff259a56ac6e08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cefe84bdcb846968ff259a56ac6e08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } } }
I20260812 11:20:19.059489 30118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:19.059526 30122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:19.060636 30128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:19.064322 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:19.051098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:59607 (local address 127.0.0.1:39941)
0812 11:20:19.051290 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:19.051295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:19.051712 (+ 417us) server_negotiation.cc:407] Connection header received
0812 11:20:19.051988 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:19.051994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:19.052061 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:19.052194 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:19.053479 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:19.054385 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:19.055709 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:19.056103 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:19.056223 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:19.059777 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:19.059783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:19.059786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:19.059822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:19.061819 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:19.062518 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:19.062522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:19.062524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:19.062609 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:19.063046 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:19.063050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:19.063052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:19.063253 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:19.063421 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:19.064037 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 11:20:19.064147 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:20:19.067061 30128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:19.075317 30128 catalog_manager.cc:1347] Generated new cluster ID: ad481dcc3d57431eae7b3a948abe9d9d
I20260812 11:20:19.075409 30128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:19.095697 30128 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:19.096977 30128 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:19.109740 30128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cefe84bdcb846968ff259a56ac6e08d: Generated new TSK 0
I20260812 11:20:19.110680 30128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:19.119060 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30004
W20260812 11:20:19.137495 29984 connection.cc:535] client connection to 127.0.0.1:39941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39941): Connection reset by peer (error 104)
2026-08-12T11:20:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:19 dist-test-slave-7z32 krb5kdc[30151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:19 dist-test-slave-7z32 krb5kdc[30151](info): set up 2 sockets
Aug 12 11:20:19 dist-test-slave-7z32 krb5kdc[30151](info): commencing operation
krb5kdc: starting...
W20260812 11:20:20.506647 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.352s user 0.003s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:20 dist-test-slave-7z32 krb5kdc[30151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:20.685125 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:20.836442 30171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:20.836884 30171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:20.836937 30171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:20.841759 30171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:20.841871 30171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:20.841893 30171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:20.841912 30171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:20.841929 30171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:20.848661 30171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:20.850322 30171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:20.851774 30171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:20.863459 30187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:20.863461 30180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:20.863492 30182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:20.864159 30171 server_base.cc:1034] running on GCE node
I20260812 11:20:20.865058 30171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:20.866699 30171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:20.867926 30171 hybrid_clock.cc:648] HybridClock initialized: now 1786533620867912 us; error 43 us; skew 500 ppm
Aug 12 11:20:20 dist-test-slave-7z32 krb5kdc[30151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:20.879769 30171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:20.881246 30171 webserver.cc:460] Webserver started at http://127.0.0.1:41129/ using document root <none> and password file <none>
I20260812 11:20:20.882035 30171 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:20.882149 30171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:20.882453 30171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:20.885375 30171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "4cbf055be5e14507af3824ab1572727d"
format_stamp: "Formatted at 2026-08-12 11:20:20 on dist-test-slave-7z32"
server_key: "9f860ffd31b1c5bb8af315599f552c56"
server_key_iv: "556ca580b3a48dc4e346f385d3e70ca4"
server_key_version: "encryptionkey@0"
I20260812 11:20:20.886188 30171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "4cbf055be5e14507af3824ab1572727d"
format_stamp: "Formatted at 2026-08-12 11:20:20 on dist-test-slave-7z32"
server_key: "9f860ffd31b1c5bb8af315599f552c56"
server_key_iv: "556ca580b3a48dc4e346f385d3e70ca4"
server_key_version: "encryptionkey@0"
I20260812 11:20:20.891752 30171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:20:20.895772 30200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:20.897794 30171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:20:20.898109 30171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "4cbf055be5e14507af3824ab1572727d"
format_stamp: "Formatted at 2026-08-12 11:20:20 on dist-test-slave-7z32"
server_key: "9f860ffd31b1c5bb8af315599f552c56"
server_key_iv: "556ca580b3a48dc4e346f385d3e70ca4"
server_key_version: "encryptionkey@0"
I20260812 11:20:20.898334 30171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:20.919551 30171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:20.924125 30171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:20.924425 30171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:20.934535 30171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43647
I20260812 11:20:20.934597 30272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43647 every 8 connection(s)
I20260812 11:20:20.936010 30171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:20.939215 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30171
I20260812 11:20:20.939411 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:20.939857 28084 external_mini_cluster.cc:1428] Setting key b5ac25d71b9bef91a0d93f73b57f067c
I20260812 11:20:20.940086 30273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:20.949400 30273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d: Bootstrap starting.
I20260812 11:20:20.952559 30273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:20.953627 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:20.942043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:32993 (local address 127.0.0.1:43647)
0812 11:20:20.942699 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:20.942722 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:20.942780 (+ 58us) server_negotiation.cc:407] Connection header received
0812 11:20:20.943635 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:20.943666 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:20.944241 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:20.944727 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:20.945276 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:20.947009 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:20.948346 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:20.949124 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:20.949367 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:20.951042 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:20.951050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:20.951069 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:20.951670 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:20.951763 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:20.951802 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:20.951993 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:20.952314 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 11:20:20.952492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":174,"threads_started":1}
I20260812 11:20:20.953593 30273 log.cc:824] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:20.956354 30273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d: No bootstrap required, opened a new log
I20260812 11:20:20.958595 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:20.961620 30273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf055be5e14507af3824ab1572727d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260812 11:20:20.962148 30273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:20.962244 30273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbf055be5e14507af3824ab1572727d, State: Initialized, Role: FOLLOWER
I20260812 11:20:20.962781 30273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf055be5e14507af3824ab1572727d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260812 11:20:20.963030 30273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:20.963122 30273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:20.963248 30273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:20.964468 30273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf055be5e14507af3824ab1572727d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260812 11:20:20.964919 30273 leader_election.cc:302] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbf055be5e14507af3824ab1572727d; no voters:
I20260812 11:20:20.965322 30273 leader_election.cc:288] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:20.965559 30286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:20.966009 30286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [term 1 LEADER]: Becoming Leader. State: Replica: 4cbf055be5e14507af3824ab1572727d, State: Running, Role: LEADER
I20260812 11:20:20.966567 30286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf055be5e14507af3824ab1572727d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260812 11:20:20.967039 30273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:20.968443 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:20.961808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:52229 (local address 127.0.0.1:43647)
0812 11:20:20.962016 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:20.962022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:20.962410 (+ 388us) server_negotiation.cc:407] Connection header received
0812 11:20:20.962618 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:20.962622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:20.962700 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:20.962878 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:20.963460 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:20.964144 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:20.965492 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:20.965951 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:20.966095 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:20.967671 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:20.967675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:20.967678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:20.967699 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:20.967743 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:20.967760 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:20.967844 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:20.968239 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 11:20:20.968288 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:20:20.969436 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cbf055be5e14507af3824ab1572727d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf055be5e14507af3824ab1572727d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } } }
I20260812 11:20:20.969663 30288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbf055be5e14507af3824ab1572727d. Latest consensus state: current_term: 1 leader_uuid: "4cbf055be5e14507af3824ab1572727d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf055be5e14507af3824ab1572727d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } } }
I20260812 11:20:20.970007 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:20.970007 30288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d [sys.catalog]: This master's current role is: LEADER
W20260812 11:20:20.973606 30308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:20.973773 30308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:20.974112 30310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:20.975083 30310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:20.982712 30310 catalog_manager.cc:1347] Generated new cluster ID: c59170885c7d49e9bd41ebcb3998630c
I20260812 11:20:20.982888 30310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:20.996244 30310 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:20.997686 30310 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:21.008214 30310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cbf055be5e14507af3824ab1572727d: Generated new TSK 0
I20260812 11:20:21.009166 30310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:21.016669 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30171
W20260812 11:20:21.040663 30149 connection.cc:535] client connection to 127.0.0.1:43647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43647): Connection reset by peer (error 104)
2026-08-12T11:20:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:21 dist-test-slave-7z32 krb5kdc[30320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:21 dist-test-slave-7z32 krb5kdc[30320](info): set up 2 sockets
Aug 12 11:20:21 dist-test-slave-7z32 krb5kdc[30320](info): commencing operation
krb5kdc: starting...
W20260812 11:20:22.435225 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.378s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:22 dist-test-slave-7z32 krb5kdc[30320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:20:22.604171 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:22.765231 30341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:22.765647 30341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:22.765703 30341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:22.770370 30341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:22.770478 30341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:22.770500 30341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:22.770520 30341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:22.770537 30341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:22.777213 30341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:22.778779 30341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:22.780221 30341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:22.791972 30348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:22.792034 30355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:22.791972 30349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:22.792739 30341 server_base.cc:1034] running on GCE node
I20260812 11:20:22.793488 30341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:22.794966 30341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:22.796195 30341 hybrid_clock.cc:648] HybridClock initialized: now 1786533622796153 us; error 72 us; skew 500 ppm
Aug 12 11:20:22 dist-test-slave-7z32 krb5kdc[30320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:22.800364 30341 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:20:22.801625 30341 webserver.cc:460] Webserver started at http://127.0.0.1:38535/ using document root <none> and password file <none>
I20260812 11:20:22.802338 30341 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:22.802425 30341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:22.802677 30341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:22.805092 30341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "ee42faf6f1774601b21138359a3a0de7"
format_stamp: "Formatted at 2026-08-12 11:20:22 on dist-test-slave-7z32"
server_key: "e8a03dec10b1bbf3a67113fa3db4f1f6"
server_key_iv: "74a29f23f745bf4ad626f1c6e6e79f95"
server_key_version: "encryptionkey@0"
I20260812 11:20:22.805795 30341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "ee42faf6f1774601b21138359a3a0de7"
format_stamp: "Formatted at 2026-08-12 11:20:22 on dist-test-slave-7z32"
server_key: "e8a03dec10b1bbf3a67113fa3db4f1f6"
server_key_iv: "74a29f23f745bf4ad626f1c6e6e79f95"
server_key_version: "encryptionkey@0"
I20260812 11:20:22.811033 30341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 11:20:22.814966 30367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:22.817152 30341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:20:22.817378 30341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "ee42faf6f1774601b21138359a3a0de7"
format_stamp: "Formatted at 2026-08-12 11:20:22 on dist-test-slave-7z32"
server_key: "e8a03dec10b1bbf3a67113fa3db4f1f6"
server_key_iv: "74a29f23f745bf4ad626f1c6e6e79f95"
server_key_version: "encryptionkey@0"
I20260812 11:20:22.817536 30341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:22.823542 30341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:22.827497 30341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:22.827746 30341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:22.836375 30341 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46265
I20260812 11:20:22.836438 30447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46265 every 8 connection(s)
I20260812 11:20:22.837563 30341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:22.841638 30449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:22.845713 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30341
I20260812 11:20:22.845929 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:22.846310 28084 external_mini_cluster.cc:1428] Setting key c28a17c63a9b91d98c5b39d0179edbdc
I20260812 11:20:22.850556 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7: Bootstrap starting.
I20260812 11:20:22.854159 30449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:22.854393 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:22.848651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:36799 (local address 127.0.0.1:46265)
0812 11:20:22.849473 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:22.849497 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:22.849562 (+ 65us) server_negotiation.cc:407] Connection header received
0812 11:20:22.850594 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:22.850619 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:22.851196 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:22.851672 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:22.852777 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:22.852783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:22.852798 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:22.852811 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:22.852888 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:22.852913 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:22.853114 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:22.853199 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 11:20:22.853414 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":259,"threads_started":1}
I20260812 11:20:22.855377 30449 log.cc:824] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:22.858978 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7: No bootstrap required, opened a new log
I20260812 11:20:22.859498 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:22.864812 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:22.862417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:57001 (local address 127.0.0.1:46265)
0812 11:20:22.862578 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:22.862582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:22.862740 (+ 158us) server_negotiation.cc:407] Connection header received
0812 11:20:22.862989 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:22.862992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:22.863036 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:22.863162 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:22.864282 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:22.864288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:22.864298 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:22.864312 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:22.864348 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:22.864364 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:22.864489 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:22.864581 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 11:20:22.864632 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:20:22.865051 30449 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42faf6f1774601b21138359a3a0de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260812 11:20:22.865551 30449 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:22.865667 30449 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee42faf6f1774601b21138359a3a0de7, State: Initialized, Role: FOLLOWER
I20260812 11:20:22.866273 30449 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42faf6f1774601b21138359a3a0de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260812 11:20:22.866519 30449 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:22.866628 30449 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:22.866770 30449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:22.868232 30449 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42faf6f1774601b21138359a3a0de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260812 11:20:22.868743 30449 leader_election.cc:302] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee42faf6f1774601b21138359a3a0de7; no voters:
I20260812 11:20:22.869223 30449 leader_election.cc:288] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:22.869513 30464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:22.869861 30464 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [term 1 LEADER]: Becoming Leader. State: Replica: ee42faf6f1774601b21138359a3a0de7, State: Running, Role: LEADER
I20260812 11:20:22.870298 30464 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42faf6f1774601b21138359a3a0de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260812 11:20:22.871021 30449 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:22.872815 30466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee42faf6f1774601b21138359a3a0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42faf6f1774601b21138359a3a0de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260812 11:20:22.872993 30466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:22.873324 30467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee42faf6f1774601b21138359a3a0de7. Latest consensus state: current_term: 1 leader_uuid: "ee42faf6f1774601b21138359a3a0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42faf6f1774601b21138359a3a0de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260812 11:20:22.873492 30467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:22.873850 30477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:22.877148 30477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:22.893373 30477 catalog_manager.cc:1347] Generated new cluster ID: 1036ee3d154f4d2e80767612ce254fde
I20260812 11:20:22.893476 30477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:22.913128 30477 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:22.913751 30477 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:22.923466 30477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee42faf6f1774601b21138359a3a0de7: Generated new TSK 0
I20260812 11:20:22.924430 30477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:22.928203 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30341
2026-08-12T11:20:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:23 dist-test-slave-7z32 krb5kdc[30497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:23 dist-test-slave-7z32 krb5kdc[30497](info): set up 2 sockets
Aug 12 11:20:23 dist-test-slave-7z32 krb5kdc[30497](info): commencing operation
krb5kdc: starting...
W20260812 11:20:24.383773 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.419s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:24 dist-test-slave-7z32 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:24.572739 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38019
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:24.714537 30513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:24.714958 30513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:24.715011 30513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:24.719285 30513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:24.719374 30513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:24.719396 30513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:24.719413 30513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:24.719430 30513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:24.725368 30513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:24.726598 30513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:24.727732 30513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:24.738165 30521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:24.738309 30520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:24.738247 30525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:24.738798 30513 server_base.cc:1034] running on GCE node
I20260812 11:20:24.739579 30513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:24.740886 30513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:24.744421 30513 hybrid_clock.cc:648] HybridClock initialized: now 1786533624744370 us; error 60 us; skew 500 ppm
Aug 12 11:20:24 dist-test-slave-7z32 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:24.748971 30513 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:24.750417 30513 webserver.cc:460] Webserver started at http://127.27.109.62:40205/ using document root <none> and password file <none>
I20260812 11:20:24.751236 30513 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:24.751359 30513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:24.751653 30513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:24.754247 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "49f1bf9b91c249288c02fbcb71f2bd80"
format_stamp: "Formatted at 2026-08-12 11:20:24 on dist-test-slave-7z32"
server_key: "a9c711ea44d25ba32e9f8c11d537207b"
server_key_iv: "b13162e95d6ccb619c61f9098467c179"
server_key_version: "encryptionkey@0"
I20260812 11:20:24.755110 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "49f1bf9b91c249288c02fbcb71f2bd80"
format_stamp: "Formatted at 2026-08-12 11:20:24 on dist-test-slave-7z32"
server_key: "a9c711ea44d25ba32e9f8c11d537207b"
server_key_iv: "b13162e95d6ccb619c61f9098467c179"
server_key_version: "encryptionkey@0"
I20260812 11:20:24.760381 30513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:20:24.764272 30534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:24.766285 30513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:20:24.766571 30513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "49f1bf9b91c249288c02fbcb71f2bd80"
format_stamp: "Formatted at 2026-08-12 11:20:24 on dist-test-slave-7z32"
server_key: "a9c711ea44d25ba32e9f8c11d537207b"
server_key_iv: "b13162e95d6ccb619c61f9098467c179"
server_key_version: "encryptionkey@0"
I20260812 11:20:24.766741 30513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:24.800803 30513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:24.804903 30513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:24.805160 30513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:24.815029 30513 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38019
I20260812 11:20:24.815090 30613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38019 every 8 connection(s)
I20260812 11:20:24.816437 30513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:24.816744 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30513
I20260812 11:20:24.816916 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:24.817337 28084 external_mini_cluster.cc:1428] Setting key 83ed3bc06ef8718904b5a63bff1d0a51
I20260812 11:20:24.821622 30615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 11:20:24 dist-test-slave-7z32 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:24.830909 30615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80: Bootstrap starting.
I20260812 11:20:24.834481 30615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:24.835745 30615 log.cc:824] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:24.837615 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:24.819442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49029 (local address 127.27.109.62:38019)
0812 11:20:24.820370 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:24.820399 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:24.820464 (+ 65us) server_negotiation.cc:407] Connection header received
0812 11:20:24.821305 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:24.821328 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:24.821918 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:24.822369 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:24.823446 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:24.824701 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:24.825760 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:24.826423 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:24.826656 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:24.830586 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:24.830594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:24.830616 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:24.830670 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:24.833155 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:24.833869 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:24.833875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:24.833882 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:24.834021 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:24.834400 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:24.834409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:24.834411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:24.835053 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:24.835335 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:24.835644 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 11:20:24.836013 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":676,"thread_start_us":260,"threads_started":1}
I20260812 11:20:24.839720 30615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80: No bootstrap required, opened a new log
I20260812 11:20:24.842723 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:24.845546 30615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38019 } }
I20260812 11:20:24.846045 30615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:24.846153 30615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f1bf9b91c249288c02fbcb71f2bd80, State: Initialized, Role: FOLLOWER
I20260812 11:20:24.846961 30615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38019 } }
I20260812 11:20:24.847198 30615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:24.847314 30615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:24.847481 30615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:24.849012 30615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38019 } }
I20260812 11:20:24.849668 30615 leader_election.cc:302] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f1bf9b91c249288c02fbcb71f2bd80; no voters:
I20260812 11:20:24.850278 30615 leader_election.cc:288] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:24.850593 30632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:24.851095 30632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [term 1 LEADER]: Becoming Leader. State: Replica: 49f1bf9b91c249288c02fbcb71f2bd80, State: Running, Role: LEADER
I20260812 11:20:24.851677 30632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38019 } }
I20260812 11:20:24.852407 30615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:24.853418 30634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f1bf9b91c249288c02fbcb71f2bd80. Latest consensus state: current_term: 1 leader_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38019 } } }
I20260812 11:20:24.853770 30634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:24.853981 30633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1bf9b91c249288c02fbcb71f2bd80" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38019 } } }
I20260812 11:20:24.854120 30633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:24.854595 30645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:24.858258 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:24.845522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44017 (local address 127.27.109.62:38019)
0812 11:20:24.845685 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:24.845692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:24.845983 (+ 291us) server_negotiation.cc:407] Connection header received
0812 11:20:24.846251 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:24.846257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:24.846317 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:24.846437 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:24.847473 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:24.848088 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:24.849410 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:24.849881 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:24.850086 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:24.852466 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:24.852472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:24.852475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:24.852531 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:24.854991 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:24.856110 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:24.856114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:24.856116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:24.856253 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:24.856806 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:24.856811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:24.856813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:24.857123 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:24.857322 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:24.857872 (+ 550us) server_negotiation.cc:299] Negotiation successful
0812 11:20:24.858017 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:20:24.859776 30645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:24.868408 30645 catalog_manager.cc:1347] Generated new cluster ID: 36d5bd32a8b84a0e82d85e3639a1ec82
I20260812 11:20:24.868530 30645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:24.880138 30645 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:24.881610 30645 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:24.890059 30645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f1bf9b91c249288c02fbcb71f2bd80: Generated new TSK 0
I20260812 11:20:24.891295 30645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:24.912653 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30513
W20260812 11:20:24.940898 30491 connection.cc:535] client connection to 127.27.109.62:38019 recv error: Network error: failed to read from TLS socket (remote: 127.27.109.62:38019): Connection reset by peer (error 104)
2026-08-12T11:20:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:25 dist-test-slave-7z32 krb5kdc[30668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:25 dist-test-slave-7z32 krb5kdc[30668](info): set up 2 sockets
Aug 12 11:20:25 dist-test-slave-7z32 krb5kdc[30668](info): commencing operation
krb5kdc: starting...
W20260812 11:20:26.354687 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.396s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:26 dist-test-slave-7z32 krb5kdc[30668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:26.528904 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32859
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:32859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:26.680238 30694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:26.680802 30694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:26.680920 30694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:26.685549 30694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:26.685693 30694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:26.685734 30694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:26.685772 30694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:26.685809 30694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:26.692346 30694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:32859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:26.694023 30694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:26.695456 30694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:26.707207 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:26.707194 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:26.707595 30694 server_base.cc:1034] running on GCE node
W20260812 11:20:26.707194 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:26.708587 30694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:26.710090 30694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:26.711305 30694 hybrid_clock.cc:648] HybridClock initialized: now 1786533626711284 us; error 36 us; skew 500 ppm
Aug 12 11:20:26 dist-test-slave-7z32 krb5kdc[30668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533626, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:26.721617 30694 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:26.723235 30694 webserver.cc:460] Webserver started at http://127.27.109.62:37153/ using document root <none> and password file <none>
I20260812 11:20:26.724054 30694 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:26.724165 30694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:26.724474 30694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:26.727377 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "63ac62468c884a4b877c573bf5efe48f"
format_stamp: "Formatted at 2026-08-12 11:20:26 on dist-test-slave-7z32"
server_key: "10f612f7a72c81950f756be94bd7ffe2"
server_key_iv: "a4c838e1e9f70cfdca4292776e4135a3"
server_key_version: "encryptionkey@0"
I20260812 11:20:26.728190 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "63ac62468c884a4b877c573bf5efe48f"
format_stamp: "Formatted at 2026-08-12 11:20:26 on dist-test-slave-7z32"
server_key: "10f612f7a72c81950f756be94bd7ffe2"
server_key_iv: "a4c838e1e9f70cfdca4292776e4135a3"
server_key_version: "encryptionkey@0"
I20260812 11:20:26.733834 30694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:20:26.738209 30715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:26.740388 30694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:20:26.740707 30694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "63ac62468c884a4b877c573bf5efe48f"
format_stamp: "Formatted at 2026-08-12 11:20:26 on dist-test-slave-7z32"
server_key: "10f612f7a72c81950f756be94bd7ffe2"
server_key_iv: "a4c838e1e9f70cfdca4292776e4135a3"
server_key_version: "encryptionkey@0"
I20260812 11:20:26.740921 30694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:26.757217 30694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:26.762144 30694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:26.762468 30694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:26.775094 30694 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:32859
I20260812 11:20:26.775183 30800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:32859 every 8 connection(s)
I20260812 11:20:26.776618 30694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:26.780571 30801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:26.780884 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30694
I20260812 11:20:26.781040 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:26.781363 28084 external_mini_cluster.cc:1428] Setting key 3adc38dd8d06abbf255f41c361fdd5c8
I20260812 11:20:26.788059 30801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f: Bootstrap starting.
I20260812 11:20:26.791211 30801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:26.792214 30801 log.cc:824] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:26.794503 30804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:26.783621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46891 (local address 127.27.109.62:32859)
0812 11:20:26.784294 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:26.784314 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:26.784365 (+ 51us) server_negotiation.cc:407] Connection header received
0812 11:20:26.785302 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:26.785328 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:26.785851 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:26.786273 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:26.786734 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:26.788000 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:26.789361 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:26.790417 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:26.790632 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:26.791918 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:26.791925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:26.791949 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:26.792596 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:26.792687 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:26.792726 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:26.792868 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:26.793159 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 11:20:26.793395 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":221,"threads_started":1}
I20260812 11:20:26.794761 30801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f: No bootstrap required, opened a new log
I20260812 11:20:26.798437 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:26.799098 30801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac62468c884a4b877c573bf5efe48f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32859 } }
I20260812 11:20:26.799470 30801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:26.799558 30801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ac62468c884a4b877c573bf5efe48f, State: Initialized, Role: FOLLOWER
I20260812 11:20:26.800112 30801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac62468c884a4b877c573bf5efe48f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32859 } }
I20260812 11:20:26.800314 30801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:26.800418 30801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:26.800544 30801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:26.801887 30801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac62468c884a4b877c573bf5efe48f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32859 } }
I20260812 11:20:26.802346 30801 leader_election.cc:302] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ac62468c884a4b877c573bf5efe48f; no voters:
I20260812 11:20:26.802778 30801 leader_election.cc:288] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:26.802975 30815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:26.803416 30815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [term 1 LEADER]: Becoming Leader. State: Replica: 63ac62468c884a4b877c573bf5efe48f, State: Running, Role: LEADER
I20260812 11:20:26.804059 30815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac62468c884a4b877c573bf5efe48f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32859 } }
I20260812 11:20:26.804425 30801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:26.806730 30820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ac62468c884a4b877c573bf5efe48f. Latest consensus state: current_term: 1 leader_uuid: "63ac62468c884a4b877c573bf5efe48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac62468c884a4b877c573bf5efe48f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32859 } } }
I20260812 11:20:26.806775 30817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ac62468c884a4b877c573bf5efe48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac62468c884a4b877c573bf5efe48f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32859 } } }
I20260812 11:20:26.806958 30820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:26.806958 30817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:26.807535 30804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:26.801424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48057 (local address 127.27.109.62:32859)
0812 11:20:26.801619 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:26.801626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:26.801888 (+ 262us) server_negotiation.cc:407] Connection header received
0812 11:20:26.802099 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:26.802102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:26.802197 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:26.802283 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:26.802674 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:26.803586 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:26.804844 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:26.805226 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:26.805357 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:26.806766 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:26.806773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:26.806776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:26.806795 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:26.806868 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:26.806940 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:26.807066 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:26.807313 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 11:20:26.807374 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 11:20:26.811291 30835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:26.811412 30835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:26.811523 30836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:26.812390 30836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:26.820190 30836 catalog_manager.cc:1347] Generated new cluster ID: f2e0bb6958634cdf92beceb681ad2f6a
I20260812 11:20:26.820305 30836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:26.829843 30836 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:26.831301 30836 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:26.840062 30836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ac62468c884a4b877c573bf5efe48f: Generated new TSK 0
I20260812 11:20:26.841250 30836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:26.846472 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30694
W20260812 11:20:26.874125 30666 connection.cc:535] client connection to 127.27.109.62:32859 recv error: Network error: failed to read from TLS socket (remote: 127.27.109.62:32859): Connection reset by peer (error 104)
2026-08-12T11:20:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:26 dist-test-slave-7z32 krb5kdc[30848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:26 dist-test-slave-7z32 krb5kdc[30848](info): set up 2 sockets
Aug 12 11:20:26 dist-test-slave-7z32 krb5kdc[30848](info): commencing operation
krb5kdc: starting...
W20260812 11:20:28.261471 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.371s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:28 dist-test-slave-7z32 krb5kdc[30848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:28.451449 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42901
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:42901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:28.609484 30873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:28.610054 30873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:28.610170 30873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:28.615477 30873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:28.615639 30873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:28.615724 30873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:28.615799 30873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:28.615866 30873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:28.623466 30873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:42901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:28.625254 30873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:28.626789 30873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:28.639988 30881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:28.639988 30883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:28.640102 30879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:28.640748 30873 server_base.cc:1034] running on GCE node
I20260812 11:20:28.641772 30873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:28.643617 30873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:28.644919 30873 hybrid_clock.cc:648] HybridClock initialized: now 1786533628644868 us; error 70 us; skew 500 ppm
Aug 12 11:20:28 dist-test-slave-7z32 krb5kdc[30848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:28.649498 30873 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:28.651491 30873 webserver.cc:460] Webserver started at http://127.27.109.62:43761/ using document root <none> and password file <none>
I20260812 11:20:28.652390 30873 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:28.652518 30873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:28.652855 30873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:28.655992 30873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "53929d86466b4017acb2e4f3a31dc99c"
format_stamp: "Formatted at 2026-08-12 11:20:28 on dist-test-slave-7z32"
server_key: "3a05c37d71cec3c175ae69f02950990b"
server_key_iv: "4b75f7e7c6b4cc5b17a3819cf5b810ad"
server_key_version: "encryptionkey@0"
I20260812 11:20:28.656876 30873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "53929d86466b4017acb2e4f3a31dc99c"
format_stamp: "Formatted at 2026-08-12 11:20:28 on dist-test-slave-7z32"
server_key: "3a05c37d71cec3c175ae69f02950990b"
server_key_iv: "4b75f7e7c6b4cc5b17a3819cf5b810ad"
server_key_version: "encryptionkey@0"
I20260812 11:20:28.663203 30873 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 11:20:28.667452 30898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:28.669726 30873 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:20:28.670039 30873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "53929d86466b4017acb2e4f3a31dc99c"
format_stamp: "Formatted at 2026-08-12 11:20:28 on dist-test-slave-7z32"
server_key: "3a05c37d71cec3c175ae69f02950990b"
server_key_iv: "4b75f7e7c6b4cc5b17a3819cf5b810ad"
server_key_version: "encryptionkey@0"
I20260812 11:20:28.670261 30873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:28.678442 30873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:28.683768 30873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:28.684080 30873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:28.695772 30873 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:42901
I20260812 11:20:28.695830 30969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:42901 every 8 connection(s)
I20260812 11:20:28.697382 30873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:28.701961 30970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:28.705157 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 30873
I20260812 11:20:28.705359 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:28.705647 28084 external_mini_cluster.cc:1428] Setting key 102fe9575be4e9eb5f8443da037ab321
I20260812 11:20:28.712348 30970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c: Bootstrap starting.
I20260812 11:20:28.714113 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:28.707986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37485 (local address 127.27.109.62:42901)
0812 11:20:28.708663 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:28.708689 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:28.708746 (+ 57us) server_negotiation.cc:407] Connection header received
0812 11:20:28.709734 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:28.709762 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:28.710346 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:28.711125 (+ 779us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:28.712253 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:28.712259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:28.712277 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:28.712289 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:28.712353 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:28.712378 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:28.712625 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:28.712721 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 11:20:28.712995 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":225,"threads_started":1}
I20260812 11:20:28.715636 30970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:28.716598 30970 log.cc:824] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:28.718998 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:28.719882 30970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c: No bootstrap required, opened a new log
I20260812 11:20:28.727366 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:28.724374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55007 (local address 127.27.109.62:42901)
0812 11:20:28.724578 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:28.724584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:28.724868 (+ 284us) server_negotiation.cc:407] Connection header received
0812 11:20:28.725172 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:28.725176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:28.725239 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:28.725367 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:28.726604 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:28.726609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:28.726612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:28.726629 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:28.726668 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:28.726683 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:28.726800 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:28.727036 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 11:20:28.727140 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:20:28.727408 30970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53929d86466b4017acb2e4f3a31dc99c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42901 } }
I20260812 11:20:28.727943 30970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:28.728044 30970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53929d86466b4017acb2e4f3a31dc99c, State: Initialized, Role: FOLLOWER
I20260812 11:20:28.728634 30970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53929d86466b4017acb2e4f3a31dc99c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42901 } }
I20260812 11:20:28.728827 30970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:28.728915 30970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:28.729070 30970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:28.730342 30970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53929d86466b4017acb2e4f3a31dc99c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42901 } }
I20260812 11:20:28.730806 30970 leader_election.cc:302] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53929d86466b4017acb2e4f3a31dc99c; no voters:
I20260812 11:20:28.731329 30970 leader_election.cc:288] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:28.731585 30981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:28.732036 30981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [term 1 LEADER]: Becoming Leader. State: Replica: 53929d86466b4017acb2e4f3a31dc99c, State: Running, Role: LEADER
I20260812 11:20:28.732630 30981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53929d86466b4017acb2e4f3a31dc99c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42901 } }
I20260812 11:20:28.733058 30970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:28.735493 30984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53929d86466b4017acb2e4f3a31dc99c. Latest consensus state: current_term: 1 leader_uuid: "53929d86466b4017acb2e4f3a31dc99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53929d86466b4017acb2e4f3a31dc99c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42901 } } }
I20260812 11:20:28.735556 30982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53929d86466b4017acb2e4f3a31dc99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53929d86466b4017acb2e4f3a31dc99c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42901 } } }
I20260812 11:20:28.735757 30982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:28.735817 30984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c [sys.catalog]: This master's current role is: LEADER
W20260812 11:20:28.740496 31008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:28.740617 31008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:28.740721 31007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:28.742172 31007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:28.750267 31007 catalog_manager.cc:1347] Generated new cluster ID: dcbb856a64874d48a80484c8ac2d4efc
I20260812 11:20:28.750398 31007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:28.769925 31007 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:28.770725 31007 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:28.784235 31007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53929d86466b4017acb2e4f3a31dc99c: Generated new TSK 0
I20260812 11:20:28.785218 31007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:28.792661 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 30873
2026-08-12T11:20:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:28 dist-test-slave-7z32 krb5kdc[31015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:28 dist-test-slave-7z32 krb5kdc[31015](info): set up 2 sockets
Aug 12 11:20:28 dist-test-slave-7z32 krb5kdc[31015](info): commencing operation
krb5kdc: starting...
W20260812 11:20:30.185322 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:30 dist-test-slave-7z32 krb5kdc[31015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:30.364077 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35155
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:35155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:30.529793 31033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:30.530237 31033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:30.530292 31033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:30.535601 31033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:30.535732 31033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:30.535755 31033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:30.535776 31033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:30.535794 31033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:30.543665 31033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:35155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:30.545564 31033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:30.547211 31033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:30.560318 31043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:30.560318 31045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:30.560472 31048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:30.560981 31033 server_base.cc:1034] running on GCE node
I20260812 11:20:30.561765 31033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:30.563292 31033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:30.564723 31033 hybrid_clock.cc:648] HybridClock initialized: now 1786533630564674 us; error 79 us; skew 500 ppm
Aug 12 11:20:30 dist-test-slave-7z32 krb5kdc[31015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:30.568704 31033 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:30.570001 31033 webserver.cc:460] Webserver started at http://127.27.109.62:43217/ using document root <none> and password file <none>
I20260812 11:20:30.570737 31033 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:30.570890 31033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:30.571165 31033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:30.573974 31033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "8259d4a53f044fa7a5b51fdce51cc147"
format_stamp: "Formatted at 2026-08-12 11:20:30 on dist-test-slave-7z32"
server_key: "c2d8d228224c1c0501ec0d0701da7648"
server_key_iv: "30366915397f154d7667c37ba8f25817"
server_key_version: "encryptionkey@0"
I20260812 11:20:30.574704 31033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "8259d4a53f044fa7a5b51fdce51cc147"
format_stamp: "Formatted at 2026-08-12 11:20:30 on dist-test-slave-7z32"
server_key: "c2d8d228224c1c0501ec0d0701da7648"
server_key_iv: "30366915397f154d7667c37ba8f25817"
server_key_version: "encryptionkey@0"
I20260812 11:20:30.580718 31033 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260812 11:20:30.584728 31057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:30.587323 31033 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:20:30.587563 31033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "8259d4a53f044fa7a5b51fdce51cc147"
format_stamp: "Formatted at 2026-08-12 11:20:30 on dist-test-slave-7z32"
server_key: "c2d8d228224c1c0501ec0d0701da7648"
server_key_iv: "30366915397f154d7667c37ba8f25817"
server_key_version: "encryptionkey@0"
I20260812 11:20:30.587708 31033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:30.632031 31033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:30.636365 31033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:30.636576 31033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:30.647172 31033 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:35155
I20260812 11:20:30.647217 31126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:35155 every 8 connection(s)
I20260812 11:20:30.648535 31033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:30.653026 31128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:30.658017 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31033
I20260812 11:20:30.658222 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:30.658571 28084 external_mini_cluster.cc:1428] Setting key e8f2f8020866362f2bc6272d2bf05c62
I20260812 11:20:30.661777 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147: Bootstrap starting.
I20260812 11:20:30.665778 31128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:30.667034 31128 log.cc:824] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:30.671165 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147: No bootstrap required, opened a new log
Aug 12 11:20:30 dist-test-slave-7z32 krb5kdc[31015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:30.676163 31128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8259d4a53f044fa7a5b51fdce51cc147" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35155 } }
I20260812 11:20:30.676595 31128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:30.676658 31128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8259d4a53f044fa7a5b51fdce51cc147, State: Initialized, Role: FOLLOWER
I20260812 11:20:30.677366 31128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8259d4a53f044fa7a5b51fdce51cc147" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35155 } }
I20260812 11:20:30.677577 31128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:30.677670 31128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:30.677800 31128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:30.679539 31128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8259d4a53f044fa7a5b51fdce51cc147" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35155 } }
I20260812 11:20:30.680117 31128 leader_election.cc:302] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8259d4a53f044fa7a5b51fdce51cc147; no voters:
I20260812 11:20:30.680680 31128 leader_election.cc:288] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:30.680830 31137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:30.682163 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:30.660732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46089 (local address 127.27.109.62:35155)
0812 11:20:30.661322 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:30.661343 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:30.661480 (+ 137us) server_negotiation.cc:407] Connection header received
0812 11:20:30.662509 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:30.662533 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:30.663043 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:30.663408 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:30.664725 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:30.666338 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:30.667992 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:30.668786 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:30.668976 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:30.673408 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:30.673416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:30.673439 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:30.673483 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:30.677398 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:30.678371 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:30.678378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:30.678386 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:30.678562 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:30.679017 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:30.679020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:30.679021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:30.679589 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:30.679850 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:30.680252 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 11:20:30.680670 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":199,"threads_started":1}
I20260812 11:20:30.681181 31137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [term 1 LEADER]: Becoming Leader. State: Replica: 8259d4a53f044fa7a5b51fdce51cc147, State: Running, Role: LEADER
I20260812 11:20:30.682631 31128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:30.683043 31137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8259d4a53f044fa7a5b51fdce51cc147" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35155 } }
I20260812 11:20:30.685393 31138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8259d4a53f044fa7a5b51fdce51cc147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8259d4a53f044fa7a5b51fdce51cc147" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35155 } } }
I20260812 11:20:30.685690 31138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:30.686035 31139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8259d4a53f044fa7a5b51fdce51cc147. Latest consensus state: current_term: 1 leader_uuid: "8259d4a53f044fa7a5b51fdce51cc147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8259d4a53f044fa7a5b51fdce51cc147" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35155 } } }
I20260812 11:20:30.686167 31139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:30.687820 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 11:20:30.691433 31159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:30.691622 31159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:30.691735 31157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:30.693111 31157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:30.704452 31157 catalog_manager.cc:1347] Generated new cluster ID: e2b7be5722084861bdedf97f0be6e69e
I20260812 11:20:30.704582 31157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:30.707792 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:30.692160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53175 (local address 127.27.109.62:35155)
0812 11:20:30.692426 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:30.692432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:30.692459 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:20:30.692613 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:30.692617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:30.692682 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:30.692824 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:30.694016 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:30.695185 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:30.697559 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:30.698073 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:30.698275 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:30.700647 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:30.700652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:30.700656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:30.700699 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:30.704319 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:30.705175 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:30.705180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:30.705183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:30.705750 (+ 567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:30.706428 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:30.706432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:30.706435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:30.706791 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:30.707055 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:30.707375 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 11:20:30.707564 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":167}
I20260812 11:20:30.719074 31157 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:30.720652 31157 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:30.728359 31157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8259d4a53f044fa7a5b51fdce51cc147: Generated new TSK 0
I20260812 11:20:30.729450 31157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:30.742267 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31033
2026-08-12T11:20:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:30 dist-test-slave-7z32 krb5kdc[31171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:30 dist-test-slave-7z32 krb5kdc[31171](info): set up 2 sockets
Aug 12 11:20:30 dist-test-slave-7z32 krb5kdc[31171](info): commencing operation
krb5kdc: starting...
W20260812 11:20:32.160665 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.382s user 0.003s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:32 dist-test-slave-7z32 krb5kdc[31171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:32.336097 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39381
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:32.516831 31190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:32.517381 31190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:32.517485 31190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:32.523641 31190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:32.523839 31190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:32.523881 31190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:32.523917 31190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:32.523954 31190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:32.531145 31190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:32.532871 31190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:32.534318 31190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:32.546103 31199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:32.545964 31198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:32.545964 31201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:32.546345 31190 server_base.cc:1034] running on GCE node
I20260812 11:20:32.547705 31190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:32.549188 31190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:32.550511 31190 hybrid_clock.cc:648] HybridClock initialized: now 1786533632550497 us; error 102 us; skew 500 ppm
Aug 12 11:20:32 dist-test-slave-7z32 krb5kdc[31171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:32.554548 31190 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:32.555821 31190 webserver.cc:460] Webserver started at http://127.27.109.62:43575/ using document root <none> and password file <none>
I20260812 11:20:32.556550 31190 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:32.556633 31190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:32.556883 31190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:32.559515 31190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7"
format_stamp: "Formatted at 2026-08-12 11:20:32 on dist-test-slave-7z32"
server_key: "b28f8aa6c6d5808fa5a8fc19d4be3e9f"
server_key_iv: "b73cc5e9e4dd92108799c992c8a2f2f8"
server_key_version: "encryptionkey@0"
I20260812 11:20:32.560189 31190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7"
format_stamp: "Formatted at 2026-08-12 11:20:32 on dist-test-slave-7z32"
server_key: "b28f8aa6c6d5808fa5a8fc19d4be3e9f"
server_key_iv: "b73cc5e9e4dd92108799c992c8a2f2f8"
server_key_version: "encryptionkey@0"
I20260812 11:20:32.565429 31190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:20:32.568898 31214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:32.570734 31190 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 11:20:32.571010 31190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7"
format_stamp: "Formatted at 2026-08-12 11:20:32 on dist-test-slave-7z32"
server_key: "b28f8aa6c6d5808fa5a8fc19d4be3e9f"
server_key_iv: "b73cc5e9e4dd92108799c992c8a2f2f8"
server_key_version: "encryptionkey@0"
I20260812 11:20:32.571169 31190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:32.591475 31190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:32.595650 31190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:32.595897 31190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:32.604794 31190 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39381
I20260812 11:20:32.604905 31301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39381 every 8 connection(s)
I20260812 11:20:32.606076 31190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:32.609071 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31190
I20260812 11:20:32.609212 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:32.609570 28084 external_mini_cluster.cc:1428] Setting key 98a5a08cecffaaa58f82d633fe9414b5
I20260812 11:20:32.611052 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:32.621639 31303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7: Bootstrap starting.
I20260812 11:20:32.622798 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:32.611734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43375 (local address 127.27.109.62:39381)
0812 11:20:32.612269 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:32.612294 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:32.612336 (+ 42us) server_negotiation.cc:407] Connection header received
0812 11:20:32.613069 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:32.613089 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:32.613491 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:32.613829 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:32.614541 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:32.615975 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:32.617573 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:32.618323 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:32.618530 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:32.619906 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:32.619914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:32.619935 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:32.620642 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:32.620748 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:32.620784 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:32.620912 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:32.621255 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 11:20:32.621616 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":189,"threads_started":1}
I20260812 11:20:32.625617 31303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:32.627018 31303 log.cc:824] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:32.627516 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:32.631613 31303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7: No bootstrap required, opened a new log
I20260812 11:20:32.636824 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:32.630766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35643 (local address 127.27.109.62:39381)
0812 11:20:32.631070 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:32.631076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:32.631262 (+ 186us) server_negotiation.cc:407] Connection header received
0812 11:20:32.631450 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:32.631454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:32.631525 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:32.631675 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:32.632277 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:32.633092 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:32.634087 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:32.634494 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:32.634672 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:32.636116 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:32.636122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:32.636125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:32.636148 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:32.636194 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:32.636211 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:32.636313 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:32.636524 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 11:20:32.636624 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260812 11:20:32.638082 31303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39381 } }
I20260812 11:20:32.638537 31303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:32.638638 31303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689b1b9c265c4bfabf19cbef8fe0d1f7, State: Initialized, Role: FOLLOWER
I20260812 11:20:32.639277 31303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39381 } }
I20260812 11:20:32.639430 31303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:32.639546 31303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:32.639678 31303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:32.641000 31303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39381 } }
I20260812 11:20:32.641505 31303 leader_election.cc:302] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 689b1b9c265c4bfabf19cbef8fe0d1f7; no voters:
I20260812 11:20:32.641957 31303 leader_election.cc:288] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:32.642217 31316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:32.642663 31316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [term 1 LEADER]: Becoming Leader. State: Replica: 689b1b9c265c4bfabf19cbef8fe0d1f7, State: Running, Role: LEADER
I20260812 11:20:32.643185 31316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39381 } }
I20260812 11:20:32.643798 31303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:32.645287 31318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 689b1b9c265c4bfabf19cbef8fe0d1f7. Latest consensus state: current_term: 1 leader_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39381 } } }
I20260812 11:20:32.645474 31318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:32.645860 31317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689b1b9c265c4bfabf19cbef8fe0d1f7" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39381 } } }
I20260812 11:20:32.646071 31330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:32.646090 31317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:32.650368 31330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:32.658524 31330 catalog_manager.cc:1347] Generated new cluster ID: 666a6f3f9395424aa2e6db0d6f722f53
I20260812 11:20:32.658722 31330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:32.673666 31330 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:32.675135 31330 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:32.684754 31330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 689b1b9c265c4bfabf19cbef8fe0d1f7: Generated new TSK 0
I20260812 11:20:32.685936 31330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:32.693005 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31190
2026-08-12T11:20:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:32 dist-test-slave-7z32 krb5kdc[31345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:32 dist-test-slave-7z32 krb5kdc[31345](info): set up 2 sockets
Aug 12 11:20:32 dist-test-slave-7z32 krb5kdc[31345](info): commencing operation
krb5kdc: starting...
W20260812 11:20:34.122349 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.371s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:34 dist-test-slave-7z32 krb5kdc[31345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:34.300232 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42741
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:34.438509 31367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:34.438928 31367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:34.439015 31367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:34.443367 31367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:34.443506 31367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:34.443542 31367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:34.443585 31367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:34.443622 31367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:34.450624 31367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:34.452138 31367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:34.453359 31367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:34.464191 31383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:34.464222 31375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:34.464340 31367 server_base.cc:1034] running on GCE node
W20260812 11:20:34.464207 31377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:34.465339 31367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:34.466745 31367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:34.468072 31367 hybrid_clock.cc:648] HybridClock initialized: now 1786533634468049 us; error 56 us; skew 500 ppm
Aug 12 11:20:34 dist-test-slave-7z32 krb5kdc[31345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533634, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:34.472011 31367 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:34.473309 31367 webserver.cc:460] Webserver started at http://127.27.109.62:35857/ using document root <none> and password file <none>
I20260812 11:20:34.474042 31367 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:34.474159 31367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:34.474450 31367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:34.477231 31367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "816fc7a8e3d3446180aa7b140b3fcb18"
format_stamp: "Formatted at 2026-08-12 11:20:34 on dist-test-slave-7z32"
server_key: "8115d0874fdb46928a2d554ebc31a5ea"
server_key_iv: "44a49bbeca29557f4bf53ba7797f064e"
server_key_version: "encryptionkey@0"
I20260812 11:20:34.478423 31367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "816fc7a8e3d3446180aa7b140b3fcb18"
format_stamp: "Formatted at 2026-08-12 11:20:34 on dist-test-slave-7z32"
server_key: "8115d0874fdb46928a2d554ebc31a5ea"
server_key_iv: "44a49bbeca29557f4bf53ba7797f064e"
server_key_version: "encryptionkey@0"
I20260812 11:20:34.485548 31367 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:20:34.490411 31393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:34.492812 31367 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260812 11:20:34.493124 31367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "816fc7a8e3d3446180aa7b140b3fcb18"
format_stamp: "Formatted at 2026-08-12 11:20:34 on dist-test-slave-7z32"
server_key: "8115d0874fdb46928a2d554ebc31a5ea"
server_key_iv: "44a49bbeca29557f4bf53ba7797f064e"
server_key_version: "encryptionkey@0"
I20260812 11:20:34.493343 31367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:34.500355 31367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:34.505275 31367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:34.505587 31367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:34.517329 31367 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:42741
I20260812 11:20:34.517390 31463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:42741 every 8 connection(s)
I20260812 11:20:34.518783 31367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:34.522696 31464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:34.522794 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31367
I20260812 11:20:34.522949 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:34.523308 28084 external_mini_cluster.cc:1428] Setting key ab3ffaad65f16cb8a0077f64961b8fc0
I20260812 11:20:34.531805 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:34.525847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51449 (local address 127.27.109.62:42741)
0812 11:20:34.526533 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:34.526555 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:34.526614 (+ 59us) server_negotiation.cc:407] Connection header received
0812 11:20:34.527624 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:34.527651 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:34.528084 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:34.528518 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:34.529835 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:34.529843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:34.529861 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:34.529881 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:34.529963 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:34.529988 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:34.530168 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:34.530337 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 11:20:34.530592 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":235,"threads_started":1}
I20260812 11:20:34.532518 31464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18: Bootstrap starting.
I20260812 11:20:34.535766 31464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:34.536348 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:34.536787 31464 log.cc:824] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:34.539932 31464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18: No bootstrap required, opened a new log
I20260812 11:20:34.541461 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:34.539138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49649 (local address 127.27.109.62:42741)
0812 11:20:34.539327 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:34.539332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:34.539615 (+ 283us) server_negotiation.cc:407] Connection header received
0812 11:20:34.539766 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:34.539769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:34.539813 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:34.539944 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:34.540942 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:34.540946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:34.540949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:20:34.540970 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:34.541004 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:20:34.541018 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:34.541142 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:34.541270 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 11:20:34.541324 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:20:34.544513 31464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42741 } }
I20260812 11:20:34.544924 31464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:20:34.544973 31464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fc7a8e3d3446180aa7b140b3fcb18, State: Initialized, Role: FOLLOWER
I20260812 11:20:34.545549 31464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42741 } }
I20260812 11:20:34.545708 31464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:20:34.545846 31464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:20:34.545962 31464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:34.547219 31464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42741 } }
I20260812 11:20:34.547654 31464 leader_election.cc:302] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816fc7a8e3d3446180aa7b140b3fcb18; no voters:
I20260812 11:20:34.548049 31464 leader_election.cc:288] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:20:34.548496 31483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:34.548906 31483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [term 1 LEADER]: Becoming Leader. State: Replica: 816fc7a8e3d3446180aa7b140b3fcb18, State: Running, Role: LEADER
I20260812 11:20:34.549412 31483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42741 } }
I20260812 11:20:34.549611 31464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:34.552405 31484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42741 } } }
I20260812 11:20:34.552824 31484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:34.552393 31485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816fc7a8e3d3446180aa7b140b3fcb18. Latest consensus state: current_term: 1 leader_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816fc7a8e3d3446180aa7b140b3fcb18" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42741 } } }
I20260812 11:20:34.552927 31485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18 [sys.catalog]: This master's current role is: LEADER
W20260812 11:20:34.556823 31501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:34.556941 31501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:34.557050 31502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:34.558068 31502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:34.566533 31502 catalog_manager.cc:1347] Generated new cluster ID: c5f366c4cd274cfd92b0435cbc02b9de
I20260812 11:20:34.566669 31502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:34.587218 31502 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:34.587677 31502 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:34.601792 31502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816fc7a8e3d3446180aa7b140b3fcb18: Generated new TSK 0
I20260812 11:20:34.603080 31502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:20:34.613971 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31367
2026-08-12T11:20:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1912 ms)
[----------] 18 tests from AuthTokenIssuingTest (34727 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:34 dist-test-slave-7z32 krb5kdc[31511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:34 dist-test-slave-7z32 krb5kdc[31511](info): set up 2 sockets
Aug 12 11:20:34 dist-test-slave-7z32 krb5kdc[31511](info): commencing operation
krb5kdc: starting...
W20260812 11:20:36.019691 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:36.187594 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33559
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33559,127.27.109.61:36683,127.27.109.60:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:36.338490 31531 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:20:36.339025 31531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:36.339087 31531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:36.339174 31531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:36.343973 31531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:36.344098 31531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:36.344121 31531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:36.344141 31531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:36.344156 31531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:36.351218 31531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33559,127.27.109.61:36683,127.27.109.60:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:36.352962 31531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:36.354444 31531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:36.366551 31537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:36.366590 31544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:36.366874 31531 server_base.cc:1034] running on GCE node
W20260812 11:20:36.367216 31547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:36.368217 31531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:36.369663 31531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:36.370944 31531 hybrid_clock.cc:648] HybridClock initialized: now 1786533636370921 us; error 44 us; skew 500 ppm
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:36.375196 31531 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:36.376451 31531 webserver.cc:460] Webserver started at http://127.27.109.62:33859/ using document root <none> and password file <none>
I20260812 11:20:36.377166 31531 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:36.377238 31531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:36.377501 31531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:36.380129 31531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "0f7b7c6aef70470d90e68115aa40dad4"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "2819802dee306eb0f3672196baf34741"
server_key_iv: "c2234782d41b29fc0f8f1c94fde6d37f"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.381054 31531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "0f7b7c6aef70470d90e68115aa40dad4"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "2819802dee306eb0f3672196baf34741"
server_key_iv: "c2234782d41b29fc0f8f1c94fde6d37f"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.386526 31531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:20:36.390273 31554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:36.392748 31531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:20:36.393121 31531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "0f7b7c6aef70470d90e68115aa40dad4"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "2819802dee306eb0f3672196baf34741"
server_key_iv: "c2234782d41b29fc0f8f1c94fde6d37f"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.393392 31531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:36.425241 31531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:36.429658 31531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:36.429955 31531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:36.439495 31531 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:33559
I20260812 11:20:36.439869 31629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:33559 every 8 connection(s)
I20260812 11:20:36.441035 31531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:36.445312 31630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:36.445914 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31531
I20260812 11:20:36.446065 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:36.446439 28084 external_mini_cluster.cc:1428] Setting key 0233aa07c41a449ad94d0bbc90d96d6b
I20260812 11:20:36.452165 31630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.61@KRBTEST.COM" created.
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:36.473929 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.455754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33559 (local address 127.0.0.1:38454)
0812 11:20:36.456533 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.456580 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:36.457702 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:36.458585 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:36.458591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:36.459231 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:36.459521 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.459536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.460984 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.460987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.461944 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.461957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.463197 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.463202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.463385 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.464113 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:36.464135 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:36.466396 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:36.469558 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.469576 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.469590 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.469972 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.470553 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.470556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.470558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.470724 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.471807 (+ 1083us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:36.471814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:36.472017 (+ 203us) client_negotiation.cc:769] Sending connection context
0812 11:20:36.472223 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 11:20:36.472610 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":7,"thread_start_us":262,"threads_started":1}
I20260812 11:20:36.473941 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.455753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.27.109.62:33559)
0812 11:20:36.456373 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:36.456397 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:36.456735 (+ 338us) server_negotiation.cc:407] Connection header received
0812 11:20:36.458120 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:36.458125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:36.458357 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:36.458584 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:36.459702 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.460811 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.462166 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.462929 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.463201 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.466647 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:36.466659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:36.466682 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:36.466726 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:36.469347 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.470156 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.470166 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.470172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.470390 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.470894 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.470899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.470903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.471647 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:36.471847 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:36.472481 (+ 634us) server_negotiation.cc:299] Negotiation successful
0812 11:20:36.472779 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":200,"threads_started":1}
I20260812 11:20:36.477358 31630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:20:36.478478 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.478064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51574)
0812 11:20:36.478287 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.478389 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260812 11:20:36.478968 31558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
I20260812 11:20:36.480254 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.479969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51584)
0812 11:20:36.480100 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.480185 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:20:36.480403 31630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:36683: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
I20260812 11:20:36.487404 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:36683
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33559,127.27.109.61:36683,127.27.109.60:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:20:36.529860 31630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } attempt: 1
I20260812 11:20:36.531385 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.530779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51590)
0812 11:20:36.531126 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.531246 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260812 11:20:36.532586 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.532302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51604)
0812 11:20:36.532447 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.532516 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 11:20:36.532842 31630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:36683: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
I20260812 11:20:36.601114 31630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } attempt: 2
I20260812 11:20:36.602480 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.601990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51608)
0812 11:20:36.602245 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.602355 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 11:20:36.603673 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.603356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51616)
0812 11:20:36.603514 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.603588 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 11:20:36.603907 31630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:36683: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
W20260812 11:20:36.621474 31643 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:20:36.621863 31643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:36.621891 31643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:36.621979 31643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:36.626945 31643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:36.627054 31643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:36.627075 31643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:36.627094 31643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:36.627111 31643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:36.633396 31643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33559,127.27.109.61:36683,127.27.109.60:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:36.634714 31643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:36.635931 31643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:36.646215 31651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:36.646215 31653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:36.646569 31650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:36.646782 31643 server_base.cc:1034] running on GCE node
I20260812 11:20:36.647543 31643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:36.648859 31643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:36.650200 31643 hybrid_clock.cc:648] HybridClock initialized: now 1786533636650171 us; error 53 us; skew 500 ppm
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:36.653915 31643 init.cc:377] Logged in from keytab as kudu/127.27.109.61@KRBTEST.COM (short username kudu)
I20260812 11:20:36.655087 31643 webserver.cc:460] Webserver started at http://127.27.109.61:44699/ using document root <none> and password file <none>
I20260812 11:20:36.655731 31643 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:36.655843 31643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:36.656111 31643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:36.658500 31643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data/instance:
uuid: "3751eee204cb44638da0fc9b954a83bf"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "4516a190d2cffbdfce8100f97d5b4d9e"
server_key_iv: "f449a84a2270119b1307d2552036ceb2"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.659225 31643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/wal/instance:
uuid: "3751eee204cb44638da0fc9b954a83bf"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "4516a190d2cffbdfce8100f97d5b4d9e"
server_key_iv: "f449a84a2270119b1307d2552036ceb2"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.664424 31643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:20:36.668197 31664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:36.670524 31643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:20:36.670770 31643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/wal
uuid: "3751eee204cb44638da0fc9b954a83bf"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "4516a190d2cffbdfce8100f97d5b4d9e"
server_key_iv: "f449a84a2270119b1307d2552036ceb2"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.671037 31643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:36.692577 31643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:36.695098 31630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } attempt: 3
I20260812 11:20:36.696343 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.695917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51624)
0812 11:20:36.696163 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.696251 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 11:20:36.697340 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.697069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51628)
0812 11:20:36.697220 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.697277 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 11:20:36.697623 31630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:36683: Network error: Client connection negotiation failed: client connection to 127.27.109.61:36683: connect: Connection refused (error 111)
I20260812 11:20:36.698102 31643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:36.698407 31643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:36.708372 31643 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:36683
I20260812 11:20:36.708743 31744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:36683 every 8 connection(s)
I20260812 11:20:36.709744 31643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
I20260812 11:20:36.713490 31745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:36.717815 31745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:20:36.718670 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31643
I20260812 11:20:36.718902 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-1/wal/instance
I20260812 11:20:36.719296 28084 external_mini_cluster.cc:1428] Setting key 6f3c8bbaf8e5d1f5e4ab2ad3577167b4
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.60@KRBTEST.COM" created.
I20260812 11:20:36.737824 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.720279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.27.109.62:33559)
0812 11:20:36.720568 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:36.720576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:36.721421 (+ 845us) server_negotiation.cc:407] Connection header received
0812 11:20:36.722959 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:36.722965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:36.723052 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:36.723180 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:36.724941 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.725864 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.727458 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.728071 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.728369 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.731951 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:36.731958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:36.731965 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:36.732031 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:36.734223 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.735474 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.735478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.735479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.735586 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.736057 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.736064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.736068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.736449 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:36.736581 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:36.737496 (+ 915us) server_negotiation.cc:299] Negotiation successful
0812 11:20:36.737661 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 11:20:36.739140 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.720589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33559 (local address 127.0.0.1:38464)
0812 11:20:36.721198 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.721257 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:36.722534 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:36.723439 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:36.723451 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:36.724031 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:36.724702 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.724718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.726117 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.726120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.727264 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.727278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.728387 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.728390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.728567 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.729346 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:36.729366 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:36.731655 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:36.734594 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.734601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.734623 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.735338 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.735701 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.735703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.735704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.735921 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.736635 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:36.736643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:36.737083 (+ 440us) client_negotiation.cc:769] Sending connection context
0812 11:20:36.737394 (+ 311us) client_negotiation.cc:240] Negotiation successful
0812 11:20:36.737849 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":189,"threads_started":1}
I20260812 11:20:36.740988 31745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:20:36.753436 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.741639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51630)
0812 11:20:36.741759 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.741776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:36.741945 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:36.742929 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:36.742932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:36.743149 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:36.743241 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.743248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.744286 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.744289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.744850 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.744859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.745533 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.745536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.745659 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.746207 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:36.746243 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:36.748114 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:36.751040 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.751047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.751051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.751434 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.751832 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.751835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.751836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.751928 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.752764 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:36.752767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:36.752893 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 11:20:36.752995 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 11:20:36.753252 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 11:20:36.753675 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.741841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.27.109.61:36683)
0812 11:20:36.742475 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:36.742497 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:36.742520 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:20:36.742557 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:36.742561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:36.742722 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:36.742935 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:36.743368 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.744155 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.744977 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.745371 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.745515 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.748427 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:36.748438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:36.748454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:36.748482 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:36.750782 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.751606 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.751612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.751614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.751719 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.752107 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.752111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.752113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.752633 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:36.752778 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:36.753176 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 11:20:36.753392 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":289,"threads_started":1}
I20260812 11:20:36.756435 31745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:20:36.757473 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.756994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59276)
0812 11:20:36.757193 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.757316 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
W20260812 11:20:36.757881 31672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
I20260812 11:20:36.759357 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.758976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59288)
0812 11:20:36.759166 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.759266 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 11:20:36.759408 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:34515
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33559,127.27.109.61:36683,127.27.109.60:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:36.759987 31745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:34515: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
I20260812 11:20:36.809394 31745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } attempt: 1
I20260812 11:20:36.810427 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.810095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59296)
0812 11:20:36.810273 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.810344 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 11:20:36.811769 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.811479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59308)
0812 11:20:36.811629 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.811692 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 11:20:36.812076 31745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:34515: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
I20260812 11:20:36.840863 31630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } attempt: 4
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:20:36.853338 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.841652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51646)
0812 11:20:36.841943 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.841961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:36.842111 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:36.842419 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:36.842422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:36.842655 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:36.842782 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.842788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.844245 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.844252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.844823 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:36.844832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.845367 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.845372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:36.845492 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.846073 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:36.846086 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:36.848240 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:36.850685 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.850696 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.850699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.851273 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.851715 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:36.851719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:36.851721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:36.851851 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:36.852535 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:36.852541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:36.852726 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 11:20:36.852876 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 11:20:36.853104 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260812 11:20:36.853380 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.841734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.27.109.61:36683)
0812 11:20:36.841899 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:36.841904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:36.842060 (+ 156us) server_negotiation.cc:407] Connection header received
0812 11:20:36.842233 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:36.842236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:36.842293 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:36.842397 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:36.842961 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.844112 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.844949 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:36.845220 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:36.845337 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:36.848499 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:36.848504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:36.848507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:36.848549 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:36.850455 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.851463 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.851466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.851468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.851574 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:36.852050 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:36.852054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:36.852056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:36.852396 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:36.852506 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:36.853019 (+ 513us) server_negotiation.cc:299] Negotiation successful
0812 11:20:36.853173 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:20:36.855062 31630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:20:36.856184 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.855729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59324)
0812 11:20:36.855947 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.856047 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 11:20:36.857161 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.856886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59328)
0812 11:20:36.857008 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.857078 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 11:20:36.857429 31630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:34515: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
I20260812 11:20:36.880357 31745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } attempt: 2
I20260812 11:20:36.881409 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.881061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59334)
0812 11:20:36.881234 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.881320 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 11:20:36.882299 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.882050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59346)
0812 11:20:36.882172 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.882231 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 11:20:36.882452 31745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:34515: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
I20260812 11:20:36.916662 31630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } attempt: 1
I20260812 11:20:36.918083 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.917546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59348)
0812 11:20:36.917781 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.917871 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260812 11:20:36.918792 31758 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260812 11:20:36.919202 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.918951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59360)
0812 11:20:36.919073 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.919142 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 11:20:36.919360 31758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:36.919430 31758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:36.919510 31758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:36.919512 31630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:34515: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
W20260812 11:20:36.924878 31758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:36.925015 31758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:36.925041 31758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:36.925063 31758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:36.925083 31758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:36.932520 31758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33559,127.27.109.61:36683,127.27.109.60:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:36.934245 31758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:36.935863 31758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:36.948266 31764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:36.948328 31771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:36.948285 31768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:36.948701 31758 server_base.cc:1034] running on GCE node
I20260812 11:20:36.949617 31758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:36.951583 31758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:36.955122 31758 hybrid_clock.cc:648] HybridClock initialized: now 1786533636955033 us; error 84 us; skew 500 ppm
Aug 12 11:20:36 dist-test-slave-7z32 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:36.960423 31758 init.cc:377] Logged in from keytab as kudu/127.27.109.60@KRBTEST.COM (short username kudu)
I20260812 11:20:36.962119 31758 webserver.cc:460] Webserver started at http://127.27.109.60:39907/ using document root <none> and password file <none>
I20260812 11:20:36.963436 31758 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:36.963598 31758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:36.963940 31758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:36.967453 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data/instance:
uuid: "a17e58b093b04896a17446f5b76bf21a"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "b5cfa48fa6d79f16c9ca6d546279f9ff"
server_key_iv: "c1807cbf7ebfe601c3374feff07dd128"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.968394 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/wal/instance:
uuid: "a17e58b093b04896a17446f5b76bf21a"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "b5cfa48fa6d79f16c9ca6d546279f9ff"
server_key_iv: "c1807cbf7ebfe601c3374feff07dd128"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.973676 31745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } attempt: 3
I20260812 11:20:36.974705 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.974414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59368)
0812 11:20:36.974556 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.974624 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 11:20:36.975137 31758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 11:20:36.976094 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.975712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59382)
0812 11:20:36.975967 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.976030 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260812 11:20:36.976343 31745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:34515: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
I20260812 11:20:36.979719 31783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:36.981915 31758 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:20:36.982221 31758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/wal
uuid: "a17e58b093b04896a17446f5b76bf21a"
format_stamp: "Formatted at 2026-08-12 11:20:36 on dist-test-slave-7z32"
server_key: "b5cfa48fa6d79f16c9ca6d546279f9ff"
server_key_iv: "c1807cbf7ebfe601c3374feff07dd128"
server_key_version: "encryptionkey@0"
I20260812 11:20:36.982432 31758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:36.986752 31630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } attempt: 2
I20260812 11:20:36.988195 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.987719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59396)
0812 11:20:36.987986 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.988090 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 11:20:36.989142 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:36.988915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59404)
0812 11:20:36.989028 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:36.989088 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 11:20:36.989281 31630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:34515: Network error: Client connection negotiation failed: client connection to 127.27.109.60:34515: connect: Connection refused (error 111)
I20260812 11:20:37.014791 31758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:37.019479 31758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:37.019789 31758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:37.030462 31758 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:34515
I20260812 11:20:37.030936 31857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:34515 every 8 connection(s)
I20260812 11:20:37.031975 31758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
I20260812 11:20:37.032644 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 31758
I20260812 11:20:37.032855 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533282802791-28084-0/minicluster-data/master-2/wal/instance
I20260812 11:20:37.033224 28084 external_mini_cluster.cc:1428] Setting key 9fe58ea58cfdb53ce3e0477e4853d3d5
I20260812 11:20:37.036867 31858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:37.043236 31858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:37.054497 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.035461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.27.109.60:34515)
0812 11:20:37.036315 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.036343 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.036477 (+ 134us) server_negotiation.cc:407] Connection header received
0812 11:20:37.037615 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.037649 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.038207 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.038717 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.039788 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.041485 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.042878 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.043624 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.043914 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.047950 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.047959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.047972 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.048009 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.050248 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.050901 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.050907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.050914 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.051059 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.051419 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.051422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.051424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.051955 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.052236 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.052516 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.052894 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":173,"threads_started":1}
I20260812 11:20:37.060741 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.046168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.27.109.62:33559)
0812 11:20:37.046404 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.046410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.046927 (+ 517us) server_negotiation.cc:407] Connection header received
0812 11:20:37.047236 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.047239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.047303 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.047468 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.048480 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.049296 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.050622 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.050960 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.051107 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.054893 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.054899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.054902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.054944 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.057973 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.058971 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.058977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.058979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.059093 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.059454 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.059460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.059462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.059729 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.059864 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.060332 (+ 468us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.060496 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:20:37.060758 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.046101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33559 (local address 127.0.0.1:38476)
0812 11:20:37.046705 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:37.046800 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:37.047060 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:37.047544 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:37.047551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:37.048112 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:37.048322 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.048335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.049512 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.049516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.050423 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.050442 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.051125 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.051129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.051299 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.052181 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:37.052225 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:37.054547 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:37.058275 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.058299 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.058313 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.058756 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.059245 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.059248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.059250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.059345 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.059865 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:37.059876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:37.060043 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 11:20:37.060245 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 11:20:37.060527 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260812 11:20:37.063012 31858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:20:37.070137 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.058170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.27.109.62:33559)
0812 11:20:37.058689 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.058703 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.058717 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:20:37.058760 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.058763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.058805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.058993 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.059763 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.060604 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.061722 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.062035 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.062156 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.065608 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.065612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.065615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.065649 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.067820 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.068385 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.068389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.068391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.068505 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.068955 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.068959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.068961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.069270 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.069550 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.069741 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.069901 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
I20260812 11:20:37.074620 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.064027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.27.109.61:36683)
0812 11:20:37.064187 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.064195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.064266 (+ 71us) server_negotiation.cc:407] Connection header received
0812 11:20:37.064459 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.064463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.064516 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.064618 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.065173 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.065760 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.066677 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.067028 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.067154 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.069765 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.069769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.069772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.069802 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.072040 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.072818 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.072824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.072826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.072989 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.073400 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.073405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.073407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.073750 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.073876 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.074298 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.074465 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260812 11:20:37.074697 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.063818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:36683 (local address 127.0.0.1:51648)
0812 11:20:37.064186 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:37.064216 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:37.064347 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:37.064635 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:37.064638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:37.064875 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:37.065031 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.065043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.065886 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.065892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.066545 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.066554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.067158 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.067162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.067267 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.067823 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:37.067836 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:37.069520 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:37.072237 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.072242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.072245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.072614 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.073156 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.073161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.073164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.073285 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.073878 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:37.073883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:37.074007 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 11:20:37.074144 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 11:20:37.074395 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260812 11:20:37.075901 31858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260812 11:20:37.086642 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.072150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.27.109.61:36683)
0812 11:20:37.072587 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.072596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.072611 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:20:37.072692 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.072696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.072840 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.072971 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.074153 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.075221 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.076408 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.076820 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.076954 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.081182 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.081186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.081189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.081224 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.083775 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.084455 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.084460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.084463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.084578 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.085163 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.085165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.085166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.085417 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.085744 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.086144 (+ 400us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.086363 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":182,"threads_started":1}
I20260812 11:20:37.089046 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.076629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59412)
0812 11:20:37.076897 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:37.076917 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:37.077003 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:37.077392 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:37.077398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:37.077661 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:37.077800 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.077810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.079751 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.079756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.080565 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.080577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.081415 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.081419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.081523 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.082485 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:37.082512 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:37.084328 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:37.086568 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.086572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.086574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.086886 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.087357 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.087361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.087363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.087489 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.088221 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:37.088224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:37.088378 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 11:20:37.088543 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 11:20:37.088736 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260812 11:20:37.089479 31630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } attempt: 3
I20260812 11:20:37.089267 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.076780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.27.109.60:34515)
0812 11:20:37.076938 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.076944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.076961 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:20:37.077125 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.077129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.077200 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.077337 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.077974 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.079536 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.080786 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.081202 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.081361 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.084555 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.084560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.084562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.084591 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.086428 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.087039 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.087044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.087046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.087185 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.087658 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.087662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.087663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.088087 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.088223 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.088714 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.089015 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:20:37.094442 31858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: Bootstrap starting.
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260812 11:20:37.096817 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:37.099079 31858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:37.100528 31858 log.cc:824] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:37.099198 31591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0f7b7c6aef70470d90e68115aa40dad4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486
I20260812 11:20:37.101616 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.090135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59420)
0812 11:20:37.090359 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:37.090379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:37.090494 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:37.091033 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:37.091037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:37.091229 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:37.091348 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.091354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.092412 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.092415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.093034 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.093044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.093636 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.093639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.093715 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.094510 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:37.094523 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:37.096282 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:37.099574 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.099579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.099581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.099865 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.100211 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.100216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.100218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.100327 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.100919 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:37.100922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:37.101046 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 11:20:37.101187 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 11:20:37.101395 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 11:20:37.101616 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.090583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.27.109.60:34515)
0812 11:20:37.090717 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.090721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.090736 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:20:37.090775 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.090778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.090888 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.090980 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.091476 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.092230 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.093166 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.093480 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.093604 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.096485 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.096491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.096495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.096535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.099419 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.100001 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.100005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.100007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.100103 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.100435 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.100440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.100442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.100774 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.100925 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.101253 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.101408 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:20:37.105217 31858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: No bootstrap required, opened a new log
I20260812 11:20:37.106457 31630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4: Bootstrap starting.
I20260812 11:20:37.110626 31630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:37.110641 31858 raft_consensus.cc:348] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
I20260812 11:20:37.111083 31858 raft_consensus.cc:729] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17e58b093b04896a17446f5b76bf21a, State: Initialized, Role: FOLLOWER
I20260812 11:20:37.111552 31858 consensus_queue.cc:260] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
I20260812 11:20:37.111987 31630 log.cc:824] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:37.112443 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.112725 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:37.113523 31858 sys_catalog.cc:564] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:37.115353 31630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4: No bootstrap required, opened a new log
I20260812 11:20:37.119467 31630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
I20260812 11:20:37.119581 31745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } attempt: 4
I20260812 11:20:37.119760 31630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7b7c6aef70470d90e68115aa40dad4, State: Initialized, Role: FOLLOWER
I20260812 11:20:37.120162 31630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
W20260812 11:20:37.120415 31892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:37.120534 31892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:37.122159 31630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:37.122951 31591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 11:20:37.123142 31591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:37.125617 31591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
Aug 12 11:20:37 dist-test-slave-7z32 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260812 11:20:37.119050 31758 master_runner.cc:186] Error getting master registration for 127.27.109.62:33559: OK, instance_id { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" instance_seqno: 1786533636436504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:20:37.128362 31758 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:20:37.128857 31895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.129029 31895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:37.129292 31591 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [CANDIDATE]: Term 1 election: Requested vote from peers 3751eee204cb44638da0fc9b954a83bf (127.27.109.61:36683), a17e58b093b04896a17446f5b76bf21a (127.27.109.60:34515)
I20260812 11:20:37.130164 31824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f7b7c6aef70470d90e68115aa40dad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a17e58b093b04896a17446f5b76bf21a"
I20260812 11:20:37.130473 31824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:37.132231 31824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f7b7c6aef70470d90e68115aa40dad4 in term 1.
W20260812 11:20:37.133888 31911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:37.133996 31911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:37.134552 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.120196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:34515 (local address 127.0.0.1:59432)
0812 11:20:37.120616 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:37.120633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:37.120733 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:37.121346 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:37.121349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:37.121589 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:37.121718 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.121725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.122763 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.122768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.123606 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:37.123619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.124355 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.124360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:37.124443 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.125062 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:37.125078 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:37.126966 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:37.129596 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.129601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.129604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.129916 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.132602 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:37.132605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:37.132607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:37.132733 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:37.133563 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:37.133567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:37.133745 (+ 178us) client_negotiation.cc:769] Sending connection context
0812 11:20:37.133913 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 11:20:37.134236 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260812 11:20:37.135182 31559 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f7b7c6aef70470d90e68115aa40dad4, a17e58b093b04896a17446f5b76bf21a; no voters:
I20260812 11:20:37.132464 31531 master_runner.cc:186] Error getting master registration for 127.27.109.61:36683: OK, instance_id { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" instance_seqno: 1786533636705439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:20:37.135612 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.120865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.27.109.60:34515)
0812 11:20:37.121008 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.121012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.121027 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:20:37.121074 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.121077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.121123 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.121208 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.121872 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.122623 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.123756 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.124100 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.124207 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.127228 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.127233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.127236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.127272 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.129446 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.132170 (+ 2724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.132175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.132177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.132434 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.132908 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.132911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.132913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.133373 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.134760 (+ 1387us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.135061 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.135324 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:20:37.135643 31895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:37.136193 31895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [term 1 LEADER]: Becoming Leader. State: Replica: 0f7b7c6aef70470d90e68115aa40dad4, State: Running, Role: LEADER
I20260812 11:20:37.136638 31895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
I20260812 11:20:37.139037 31745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: Bootstrap starting.
I20260812 11:20:37.139117 31918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7b7c6aef70470d90e68115aa40dad4. Latest consensus state: current_term: 1 leader_uuid: "0f7b7c6aef70470d90e68115aa40dad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.139343 31918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:37.140353 31531 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:20:37.142697 31745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:37.143795 31745 log.cc:824] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:37.148627 31922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:37.149820 31745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: No bootstrap required, opened a new log
I20260812 11:20:37.151835 31922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:37.156438 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.139475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.27.109.61:36683)
0812 11:20:37.139603 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.139608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.141148 (+ 1540us) server_negotiation.cc:407] Connection header received
0812 11:20:37.141371 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.141375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.141427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.141484 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.142647 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.143515 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.145574 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.146038 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.146222 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.149674 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.149679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.149682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.149722 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.153251 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.154314 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.154319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.154321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.154431 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.155008 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.155012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.155014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.155290 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.155428 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.156043 (+ 615us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.156214 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:20:37.154656 31745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
I20260812 11:20:37.163431 31745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751eee204cb44638da0fc9b954a83bf, State: Initialized, Role: FOLLOWER
I20260812 11:20:37.164140 31745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } }
I20260812 11:20:37.165097 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.140610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.27.109.60:34515)
0812 11:20:37.140749 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.140755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.148095 (+ 7340us) server_negotiation.cc:407] Connection header received
0812 11:20:37.149761 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.149765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.149831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.149898 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.154444 (+ 4546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.155328 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.156713 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.157057 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.157144 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.159490 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.159495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.159498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.159540 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.161703 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.162539 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.162543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.162545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.162657 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.163181 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.163186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.163188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.163531 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.163695 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.164684 (+ 989us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.164872 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:20:37.166020 31745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:37.166457 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f7b7c6aef70470d90e68115aa40dad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3751eee204cb44638da0fc9b954a83bf"
I20260812 11:20:37.166937 31704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:37.168257 31824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [term 1 FOLLOWER]: Refusing update from remote peer 0f7b7c6aef70470d90e68115aa40dad4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:20:37.169147 31704 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f7b7c6aef70470d90e68115aa40dad4 in term 1.
I20260812 11:20:37.169116 31918 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:20:37.170172 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:37.140191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.27.109.62:33559)
0812 11:20:37.140311 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.140314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.147534 (+ 7220us) server_negotiation.cc:407] Connection header received
0812 11:20:37.147812 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.147816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.147900 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.147972 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.149092 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.154561 (+ 5469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.155875 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:37.156295 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:37.156386 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:37.158846 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:37.158854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:37.158860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:37.158899 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:37.161042 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.168257 (+ 7215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.168263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.168266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.168399 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:37.168883 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:37.168890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:37.168894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:37.169215 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:37.169365 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:37.169800 (+ 435us) server_negotiation.cc:299] Negotiation successful
0812 11:20:37.169981 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 11:20:37.172215 31932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.172425 31932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:37.174710 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7b7c6aef70470d90e68115aa40dad4. Latest consensus state: current_term: 1 leader_uuid: "0f7b7c6aef70470d90e68115aa40dad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.174928 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:37.176708 31704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [term 1 FOLLOWER]: Refusing update from remote peer 0f7b7c6aef70470d90e68115aa40dad4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:20:37.180408 31918 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:20:37.181401 31948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:37.181522 31948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:37.181981 31895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f7b7c6aef70470d90e68115aa40dad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.182150 31895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:37.184696 31922 catalog_manager.cc:1347] Generated new cluster ID: 0ab8b156ab144a32adf179f51a0963ea
I20260812 11:20:37.184820 31922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:37.188788 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f7b7c6aef70470d90e68115aa40dad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.188966 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:37.189755 31932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7b7c6aef70470d90e68115aa40dad4. Latest consensus state: current_term: 1 leader_uuid: "0f7b7c6aef70470d90e68115aa40dad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.189898 31932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:37.189849 31950 mvcc.cc:205] Tried to move back new op lower bound from 7317641777829429248 to 7317641777716039680. Current Snapshot: MvccSnapshot[applied={T|T < 7317641777829429248 or (T in {7317641777829429248})}]
I20260812 11:20:37.199759 31918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f7b7c6aef70470d90e68115aa40dad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.199959 31918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4 [sys.catalog]: This master's current role is: LEADER
W20260812 11:20:37.200815 31675 tablet.cc:2369] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:20:37.201406 31932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f7b7c6aef70470d90e68115aa40dad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7b7c6aef70470d90e68115aa40dad4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33559 } } peers { permanent_uuid: "3751eee204cb44638da0fc9b954a83bf" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 36683 } } peers { permanent_uuid: "a17e58b093b04896a17446f5b76bf21a" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 34515 } } }
I20260812 11:20:37.201582 31932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:37.202304 31956 mvcc.cc:205] Tried to move back new op lower bound from 7317641777829429248 to 7317641777716039680. Current Snapshot: MvccSnapshot[applied={T|T < 7317641777829429248}]
I20260812 11:20:37.228931 31922 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:37.231016 31922 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:37.243378 31922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f7b7c6aef70470d90e68115aa40dad4: Generated new TSK 0
I20260812 11:20:37.244864 31922 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:20:37.295922 31867 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.292132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.27.109.62:33559)
0812 11:20:37.292306 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.292313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.294375 (+ 2062us) server_negotiation.cc:407] Connection header received
0812 11:20:37.294634 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.294639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.294707 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.294772 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.295726 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38506: BlockingRecv error: recv got EOF from 127.0.0.1:38506 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 11:20:37.298568 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.293162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.27.109.60:34515)
0812 11:20:37.293320 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.293325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.296921 (+ 3596us) server_negotiation.cc:407] Connection header received
0812 11:20:37.297267 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.297271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.297358 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.297421 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.298389 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59442: BlockingRecv error: recv got EOF from 127.0.0.1:59442 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 11:20:37.300467 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.293560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.27.109.61:36683)
0812 11:20:37.293713 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.293718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.299176 (+ 5458us) server_negotiation.cc:407] Connection header received
0812 11:20:37.299420 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.299426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.299505 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.299572 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.300302 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51676: BlockingRecv error: recv got EOF from 127.0.0.1:51676 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 11:20:37.313174 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.307678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.27.109.61:36683)
0812 11:20:37.307863 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.307869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.311552 (+ 3683us) server_negotiation.cc:407] Connection header received
0812 11:20:37.311945 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.311950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.312077 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.312162 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.312984 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51692: BlockingRecv error: recv got EOF from 127.0.0.1:51692 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 11:20:37.322525 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.320099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.27.109.61:36683)
0812 11:20:37.320285 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.320291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.321204 (+ 913us) server_negotiation.cc:407] Connection header received
0812 11:20:37.321472 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.321477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.321547 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.321610 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.322362 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51698: BlockingRecv error: recv got EOF from 127.0.0.1:51698 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260812 11:20:37.330228 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.327959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.27.109.61:36683)
0812 11:20:37.328112 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.328117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.328905 (+ 788us) server_negotiation.cc:407] Connection header received
0812 11:20:37.329199 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.329203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.329271 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.329333 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.330050 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51708: BlockingRecv error: recv got EOF from 127.0.0.1:51708 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 11:20:37.339926 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.337629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.27.109.61:36683)
0812 11:20:37.337790 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.337795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.338545 (+ 750us) server_negotiation.cc:407] Connection header received
0812 11:20:37.338845 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.338850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.338925 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.338992 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.339765 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51710: BlockingRecv error: recv got EOF from 127.0.0.1:51710 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 11:20:37.354211 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.351781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.27.109.61:36683)
0812 11:20:37.351964 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.351969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.352808 (+ 839us) server_negotiation.cc:407] Connection header received
0812 11:20:37.353081 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.353086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.353166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.353237 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.353992 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51718: BlockingRecv error: recv got EOF from 127.0.0.1:51718 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 11:20:37.376930 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.374092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.27.109.61:36683)
0812 11:20:37.374282 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.374288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.375257 (+ 969us) server_negotiation.cc:407] Connection header received
0812 11:20:37.375593 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.375598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.375698 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.375774 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.376753 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51728: BlockingRecv error: recv got EOF from 127.0.0.1:51728 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 11:20:37.414144 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.412549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.27.109.61:36683)
0812 11:20:37.412754 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.412759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.412989 (+ 230us) server_negotiation.cc:407] Connection header received
0812 11:20:37.413211 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.413220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.413280 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.413383 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.413957 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51742: BlockingRecv error: recv got EOF from 127.0.0.1:51742 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 11:20:37.482743 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.481304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.27.109.61:36683)
0812 11:20:37.481524 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.481528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.481698 (+ 170us) server_negotiation.cc:407] Connection header received
0812 11:20:37.481798 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.481801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.481876 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.481995 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.482625 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51748: BlockingRecv error: recv got EOF from 127.0.0.1:51748 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260812 11:20:37.615824 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.614136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.27.109.61:36683)
0812 11:20:37.614441 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.614449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.614588 (+ 139us) server_negotiation.cc:407] Connection header received
0812 11:20:37.614810 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.614813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.615035 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.615141 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.615691 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51764: BlockingRecv error: recv got EOF from 127.0.0.1:51764 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260812 11:20:37.876919 31869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:37.875596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.27.109.61:36683)
0812 11:20:37.875762 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:37.875766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:37.875807 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:20:37.876041 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:37.876044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:37.876109 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:37.876224 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:37.876767 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51770: BlockingRecv error: recv got EOF from 127.0.0.1:51770 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:20:38.122668 31892 catalog_manager.cc:1259] Loaded cluster ID: 0ab8b156ab144a32adf179f51a0963ea
I20260812 11:20:38.122809 31892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: loading cluster ID for follower catalog manager: success
I20260812 11:20:38.125280 31892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: acquiring CA information for follower catalog manager: success
I20260812 11:20:38.127044 31892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a17e58b093b04896a17446f5b76bf21a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:20:38.183748 31948 catalog_manager.cc:1259] Loaded cluster ID: 0ab8b156ab144a32adf179f51a0963ea
I20260812 11:20:38.183884 31948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: loading cluster ID for follower catalog manager: success
I20260812 11:20:38.186126 31948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: acquiring CA information for follower catalog manager: success
I20260812 11:20:38.188479 31948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3751eee204cb44638da0fc9b954a83bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:20:38.396667 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:38.392002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.27.109.61:36683)
0812 11:20:38.392377 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:38.392385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:38.392406 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:20:38.392487 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:38.392490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:38.392669 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:38.392816 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:38.393129 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.393820 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.394775 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.395038 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.395160 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:38.395570 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:38.396181 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:38.396331 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:38.396380 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 11:20:38.396456 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":92,"threads_started":1}
I20260812 11:20:38.406080 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:38.401299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.27.109.60:34515)
0812 11:20:38.401642 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:38.401647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:38.401665 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:20:38.401931 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:38.401934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:38.402097 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:38.402264 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:38.402521 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.403214 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.404172 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.404423 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.404544 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:38.404923 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:38.405561 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:38.405717 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:38.405784 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 11:20:38.405864 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260812 11:20:38.418282 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:38.411924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.27.109.61:36683)
0812 11:20:38.412125 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:38.412130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:38.412899 (+ 769us) server_negotiation.cc:407] Connection header received
0812 11:20:38.413105 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:38.413108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:38.413186 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:38.413286 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:38.413871 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.414926 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.416602 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.416982 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.417066 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:38.417619 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:38.417724 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:38.417778 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:38.417999 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 11:20:38.418071 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 11:20:38.418403 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:38.412480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.27.109.60:34515)
0812 11:20:38.413181 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:38.413186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:38.413436 (+ 250us) server_negotiation.cc:407] Connection header received
0812 11:20:38.413696 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:38.413699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:38.413781 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:38.413844 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:38.414241 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.415117 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.416750 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.417136 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.417218 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:38.417619 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:38.417724 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:38.417791 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:38.418057 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 11:20:38.418184 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604}
I20260812 11:20:38.419093 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:38.412023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.27.109.62:33559)
0812 11:20:38.412438 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:38.412446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:38.412466 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:20:38.412657 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:38.412664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:38.412923 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:38.413105 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:38.413694 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.414534 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.416218 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:38.416609 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:38.416727 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:38.417343 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:38.418019 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:38.418118 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:38.418810 (+ 692us) server_negotiation.cc:299] Negotiation successful
0812 11:20:38.418928 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":87,"threads_started":1}
I20260812 11:20:38.432497 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31531
I20260812 11:20:38.460075 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31643
I20260812 11:20:38.482225 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 31758
2026-08-12T11:20:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3853 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:20:38 dist-test-slave-7z32 krb5kdc[32076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:20:38 dist-test-slave-7z32 krb5kdc[32076](info): set up 2 sockets
Aug 12 11:20:38 dist-test-slave-7z32 krb5kdc[32076](info): commencing operation
krb5kdc: starting...
W20260812 11:20:39.872018 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.355s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/test-user.keytab.
Aug 12 11:20:39 dist-test-slave-7z32 krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:20:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260812 11:20:40.055095 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45665
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45665,127.27.109.61:32789,127.27.109.60:46673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:20:40.217991 32094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:20:40.218604 32094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:40.218686 32094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:40.218792 32094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:40.224129 32094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:40.224274 32094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:40.224313 32094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:40.224359 32094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:40.224398 32094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:40.232208 32094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45665,127.27.109.61:32789,127.27.109.60:46673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:40.234040 32094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:40.235670 32094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:40.247861 32101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:40.247887 32105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:40.247985 32094 server_base.cc:1034] running on GCE node
W20260812 11:20:40.247915 32100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:40.249210 32094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:40.250735 32094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:40.252020 32094 hybrid_clock.cc:648] HybridClock initialized: now 1786533640251980 us; error 74 us; skew 500 ppm
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:40.255854 32094 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260812 11:20:40.257107 32094 webserver.cc:460] Webserver started at http://127.27.109.62:36475/ using document root <none> and password file <none>
I20260812 11:20:40.257800 32094 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:40.257901 32094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:40.258165 32094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:40.260670 32094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data/instance:
uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "19ecb81730d95ad6468c12fd80896cc0"
server_key_iv: "a4ef981eff4320d5c12b66679557bb2c"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.261345 32094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/wal/instance:
uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "19ecb81730d95ad6468c12fd80896cc0"
server_key_iv: "a4ef981eff4320d5c12b66679557bb2c"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.266485 32094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 11:20:40.270162 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:40.272329 32094 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:20:40.272584 32094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/wal
uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "19ecb81730d95ad6468c12fd80896cc0"
server_key_iv: "a4ef981eff4320d5c12b66679557bb2c"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.272758 32094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:40.303483 32094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:40.307936 32094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:40.308207 32094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:40.318781 32094 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:45665
I20260812 11:20:40.319556 32194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:45665 every 8 connection(s)
I20260812 11:20:40.320604 32094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/data/info.pb
I20260812 11:20:40.325667 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:40.328245 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 32094
I20260812 11:20:40.328447 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-0/wal/instance
I20260812 11:20:40.329018 28084 external_mini_cluster.cc:1428] Setting key 33c6923d1af370fc6ca638d7aaa346ea
I20260812 11:20:40.333639 32198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.61@KRBTEST.COM" created.
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:40.355577 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.337619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45665 (local address 127.0.0.1:52304)
0812 11:20:40.338615 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.338673 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.339953 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.340952 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.340958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.341557 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.341953 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.341973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.343591 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.343595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.344703 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.344715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.345521 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.345525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.345671 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.346620 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.346646 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.348796 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.351465 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.351476 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.351491 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.351788 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.352214 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.352219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.352222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.352366 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.353194 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.353202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.353377 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.353608 (+ 231us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.354141 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":133,"thread_start_us":252,"threads_started":1}
I20260812 11:20:40.355577 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.337617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.27.109.62:45665)
0812 11:20:40.338341 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.338368 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.338871 (+ 503us) server_negotiation.cc:407] Connection header received
0812 11:20:40.340374 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.340380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.340689 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.340952 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.342149 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.343427 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.344844 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.345312 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.345536 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.349059 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.349067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.349083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.349109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.351307 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.351946 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.351962 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.351963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.352101 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.352476 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.352482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.352484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.353050 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.353228 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.353682 (+ 454us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.354045 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":216,"threads_started":1}
I20260812 11:20:40.359547 32198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:20:40.360790 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.360309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57642)
0812 11:20:40.360539 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.360644 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260812 11:20:40.361073 32120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
I20260812 11:20:40.362522 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.362220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57658)
0812 11:20:40.362358 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.362416 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 11:20:40.362869 32198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:32789: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
I20260812 11:20:40.367290 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:32789
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45665,127.27.109.61:32789,127.27.109.60:46673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:20:40.412214 32198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } attempt: 1
I20260812 11:20:40.413836 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.413218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57666)
0812 11:20:40.413546 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.413668 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 11:20:40.415812 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.415118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57672)
0812 11:20:40.415571 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.415659 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304}
W20260812 11:20:40.416712 32198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:32789: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
I20260812 11:20:40.484954 32198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } attempt: 2
I20260812 11:20:40.486370 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.485907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57680)
0812 11:20:40.486157 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.486254 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 11:20:40.487376 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.487086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57692)
0812 11:20:40.487220 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.487286 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 11:20:40.487516 32198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:32789: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
W20260812 11:20:40.507279 32211 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:20:40.507712 32211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:40.507781 32211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:40.507885 32211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:40.512301 32211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:40.512416 32211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:40.512454 32211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:40.512494 32211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:40.512530 32211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:40.519556 32211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45665,127.27.109.61:32789,127.27.109.60:46673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:40.520969 32211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:40.522173 32211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:20:40.534044 32219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:40.534027 32223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:40.534144 32211 server_base.cc:1034] running on GCE node
W20260812 11:20:40.534027 32218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:40.535238 32211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:40.536803 32211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:40.538054 32211 hybrid_clock.cc:648] HybridClock initialized: now 1786533640538019 us; error 75 us; skew 500 ppm
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:40.541803 32211 init.cc:377] Logged in from keytab as kudu/127.27.109.61@KRBTEST.COM (short username kudu)
I20260812 11:20:40.543157 32211 webserver.cc:460] Webserver started at http://127.27.109.61:40091/ using document root <none> and password file <none>
I20260812 11:20:40.543834 32211 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:40.543949 32211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:40.544222 32211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:40.547169 32211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data/instance:
uuid: "afda5da8847d4d308733a6288b19b0c4"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "4ee88da63472131aaee3a4a22fa6a513"
server_key_iv: "51b83413309b2e41273462bbc9a9fc09"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.547950 32211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/wal/instance:
uuid: "afda5da8847d4d308733a6288b19b0c4"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "4ee88da63472131aaee3a4a22fa6a513"
server_key_iv: "51b83413309b2e41273462bbc9a9fc09"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.552670 32211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:20:40.555979 32236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:40.557624 32211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:20:40.557806 32211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/wal
uuid: "afda5da8847d4d308733a6288b19b0c4"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "4ee88da63472131aaee3a4a22fa6a513"
server_key_iv: "51b83413309b2e41273462bbc9a9fc09"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.557960 32211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:40.573994 32211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:40.578056 32211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:40.578313 32211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:40.578735 32198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } attempt: 3
I20260812 11:20:40.580096 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.579578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57702)
0812 11:20:40.579859 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.579966 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 11:20:40.581162 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.580889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57706)
0812 11:20:40.581018 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.581086 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 11:20:40.581332 32198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:32789: Network error: Client connection negotiation failed: client connection to 127.27.109.61:32789: connect: Connection refused (error 111)
I20260812 11:20:40.587713 32211 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:32789
I20260812 11:20:40.588135 32299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:32789 every 8 connection(s)
I20260812 11:20:40.589079 32211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/data/info.pb
I20260812 11:20:40.593266 32300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:40.598438 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 32211
I20260812 11:20:40.598582 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-1/wal/instance
I20260812 11:20:40.599056 28084 external_mini_cluster.cc:1428] Setting key 64c2a78c1e58393084c98e88058c8f39
I20260812 11:20:40.599290 32300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.60@KRBTEST.COM" created.
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:40.621598 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.602177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.27.109.62:45665)
0812 11:20:40.602398 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.602406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.603886 (+ 1480us) server_negotiation.cc:407] Connection header received
0812 11:20:40.605811 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.605817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.605903 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.606028 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.608192 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.608871 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.610673 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.611189 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.611335 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.615473 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.615479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.615482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.615540 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.618037 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.619057 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.619061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.619063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.619190 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.619651 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.619654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.619656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.620099 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.620294 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.621155 (+ 861us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.621368 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 11:20:40.623297 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.602642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45665 (local address 127.0.0.1:52318)
0812 11:20:40.603619 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.603722 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.605392 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.606333 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.606347 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.607043 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.607909 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.607937 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.609078 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.609082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.610497 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.610511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.611413 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.611418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.611594 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.612521 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.612547 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.615032 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.618298 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.618307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.618333 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.618804 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.619317 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.619321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.619324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.619520 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.620289 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.620300 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.620804 (+ 504us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.621060 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.621614 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":301,"threads_started":1}
I20260812 11:20:40.626082 32300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
I20260812 11:20:40.639742 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.626804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57712)
0812 11:20:40.627022 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.627042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.627149 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.628077 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.628080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.628382 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.628525 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.628540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.629666 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.629669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.630359 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.630371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.631201 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.631204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.631317 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.632089 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.632107 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.634078 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.636973 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.636980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.636983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.637233 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.637897 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.637901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.637903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.638021 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.639066 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.639068 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.639216 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.639407 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.639589 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260812 11:20:40.640053 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.627105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.27.109.61:32789)
0812 11:20:40.627515 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.627530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.627554 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:20:40.627598 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.627603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.627929 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.628099 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.628648 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.629528 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.630525 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.631066 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.631196 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.634256 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.634269 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.634288 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.634309 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.636842 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.637447 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.637457 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.637458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.637633 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.638217 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.638221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.638222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.638814 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.639090 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.639583 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.639856 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
I20260812 11:20:40.640887 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
/tmp/dist-test-taskK9g2P8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:46673
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45665,127.27.109.61:32789,127.27.109.60:46673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:20:40.643404 32300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:20:40.644630 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.644211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60538)
0812 11:20:40.644393 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.644508 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 11:20:40.645035 32238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
I20260812 11:20:40.646765 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.646284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60544)
0812 11:20:40.646524 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.646600 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260812 11:20:40.647097 32300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46673: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
I20260812 11:20:40.696641 32300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } attempt: 1
I20260812 11:20:40.698167 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.697549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60552)
0812 11:20:40.697882 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.697996 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 11:20:40.699538 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.699191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60566)
0812 11:20:40.699358 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.699446 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 11:20:40.699842 32300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46673: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
I20260812 11:20:40.724588 32198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } attempt: 4
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:20:40.741797 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.725830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57728)
0812 11:20:40.726173 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.726197 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.726404 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.727012 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.727022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.727334 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.727651 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.727663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.729007 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.729017 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.729763 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.729780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.730922 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.730926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.731182 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.731974 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.731991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.734806 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.738626 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.738631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.738633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.739110 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.739708 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.739714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.739717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.739962 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.740900 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.740905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.741103 (+ 198us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.741256 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.741524 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260812 11:20:40.742352 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.725973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.27.109.61:32789)
0812 11:20:40.726182 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.726187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.726405 (+ 218us) server_negotiation.cc:407] Connection header received
0812 11:20:40.726531 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.726534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.726605 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.726768 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.727942 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.728707 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.730187 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.730671 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.730915 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.735375 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.735379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.735382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.735429 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.738158 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.739291 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.739296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.739298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.739462 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.740271 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.740279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.740281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.740618 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.740908 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.741719 (+ 811us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.742085 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:20:40.743824 32198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:20:40.745679 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.745188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60582)
0812 11:20:40.745448 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.745553 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260812 11:20:40.746726 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.746343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60592)
0812 11:20:40.746550 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.746622 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260812 11:20:40.747040 32198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46673: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
I20260812 11:20:40.768294 32300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } attempt: 2
I20260812 11:20:40.769621 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.769168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60598)
0812 11:20:40.769441 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.769515 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260812 11:20:40.770556 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.770296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60606)
0812 11:20:40.770426 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.770486 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 11:20:40.770731 32300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46673: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
W20260812 11:20:40.785890 32308 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:20:40.786340 32308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:20:40.786412 32308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:20:40.786500 32308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:20:40.791109 32308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:20:40.791466 32308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:20:40.791560 32308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:20:40.791610 32308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:20:40.791672 32308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:20:40.798169 32308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45665,127.27.109.61:32789,127.27.109.60:46673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:46673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 10:26:14 UTC on ccycloud.quasar-hqgckh.root.comops.site
build id 4582280
I20260812 11:20:40.799587 32308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:20:40.800792 32308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:20:40.806238 32198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } attempt: 1
I20260812 11:20:40.807551 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.807089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60616)
0812 11:20:40.807301 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.807392 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 11:20:40.808456 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.808220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60620)
0812 11:20:40.808332 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.808396 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 11:20:40.808606 32198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46673: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
W20260812 11:20:40.812705 32323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:40.812870 32320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:20:40.813125 32321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:20:40.813647 32308 server_base.cc:1034] running on GCE node
I20260812 11:20:40.814994 32308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:20:40.817755 32308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:20:40.821945 32308 hybrid_clock.cc:648] HybridClock initialized: now 1786533640821459 us; error 126 us; skew 500 ppm
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:20:40.827214 32308 init.cc:377] Logged in from keytab as kudu/127.27.109.60@KRBTEST.COM (short username kudu)
I20260812 11:20:40.828953 32308 webserver.cc:460] Webserver started at http://127.27.109.60:38427/ using document root <none> and password file <none>
I20260812 11:20:40.829833 32308 fs_manager.cc:359] Metadata directory not provided
I20260812 11:20:40.829928 32308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:20:40.830228 32308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:20:40.833673 32308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data/instance:
uuid: "4c8672854f9844579a02a635edd567e9"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "cd143dd0e6b396f63232b17f3cab5ca7"
server_key_iv: "573ce4e861518600cc22e7cb018ddfc7"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.834631 32308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/wal/instance:
uuid: "4c8672854f9844579a02a635edd567e9"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "cd143dd0e6b396f63232b17f3cab5ca7"
server_key_iv: "573ce4e861518600cc22e7cb018ddfc7"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.840785 32308 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260812 11:20:40.845350 32334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:20:40.848162 32308 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 11:20:40.848475 32308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/wal
uuid: "4c8672854f9844579a02a635edd567e9"
format_stamp: "Formatted at 2026-08-12 11:20:40 on dist-test-slave-7z32"
server_key: "cd143dd0e6b396f63232b17f3cab5ca7"
server_key_iv: "573ce4e861518600cc22e7cb018ddfc7"
server_key_version: "encryptionkey@0"
I20260812 11:20:40.848671 32308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:20:40.861943 32300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } attempt: 3
I20260812 11:20:40.863292 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.862749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60636)
0812 11:20:40.863063 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.863168 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260812 11:20:40.864324 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.864001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60650)
0812 11:20:40.864193 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.864256 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260812 11:20:40.864579 32300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46673: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
I20260812 11:20:40.869962 32308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:20:40.875516 32308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:20:40.875847 32308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:20:40.875840 32198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } attempt: 2
I20260812 11:20:40.877170 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.876660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60652)
0812 11:20:40.876899 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.877018 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 11:20:40.877998 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.877762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60654)
0812 11:20:40.877877 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.877932 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:20:40.878168 32198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46673: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46673: connect: Connection refused (error 111)
I20260812 11:20:40.887439 32308 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:46673
I20260812 11:20:40.888175 32411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:46673 every 8 connection(s)
I20260812 11:20:40.889002 32308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/data/info.pb
I20260812 11:20:40.892828 32412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:20:40.893987 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu as pid 32308
I20260812 11:20:40.894308 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9g2P8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533282802791-28084-0/minicluster-data/master-2/wal/instance
I20260812 11:20:40.894729 28084 external_mini_cluster.cc:1428] Setting key e73e17facc99bcdc18189b551681768d
I20260812 11:20:40.899175 32412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:40.917091 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.901869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.27.109.62:45665)
0812 11:20:40.902268 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.902274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.902414 (+ 140us) server_negotiation.cc:407] Connection header received
0812 11:20:40.902667 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.902671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.902747 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.902973 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.904262 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.905174 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.906261 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.906638 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.906765 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.910738 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.910743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.910746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.910793 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.913272 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.914143 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.914148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.914149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.914279 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.915000 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.915005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.915007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.915750 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.915887 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.916736 (+ 849us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.916930 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260812 11:20:40.917146 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.901706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:45665 (local address 127.0.0.1:52334)
0812 11:20:40.902231 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.902296 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.902463 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.903054 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.903061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.903625 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.903814 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.903821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.905384 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.905388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.906088 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.906099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.906979 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.906984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.907125 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.908113 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.908133 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.910432 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.913437 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.913454 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.913462 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.913997 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.914469 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.914474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.914476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.914609 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.916168 (+ 1559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.916179 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.916390 (+ 211us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.916574 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.916816 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":142,"threads_started":1}
I20260812 11:20:40.917163 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.896610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.27.109.60:46673)
0812 11:20:40.897158 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.897176 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.897223 (+ 47us) server_negotiation.cc:407] Connection header received
0812 11:20:40.898034 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.898055 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.898501 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.898964 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.900008 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.901131 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.903002 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.903794 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.904032 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.908502 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.908510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.908528 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.908557 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.911043 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.912142 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.912153 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.912166 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.912404 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.912874 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.912886 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.912888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.913830 (+ 942us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.914194 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.914891 (+ 697us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.915669 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":179,"threads_started":1}
I20260812 11:20:40.919885 32412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260812 11:20:40.933691 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.920783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32789 (local address 127.0.0.1:57736)
0812 11:20:40.921022 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.921042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.921152 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.921703 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.921706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.921974 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.922182 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.922193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.923586 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.923589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.924431 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.924447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.925254 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.925295 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.925452 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.926190 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.926212 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.928279 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.931290 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.931297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.931300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.931634 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.932080 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.932084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.932086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.932258 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.932897 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.932901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.933046 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.933227 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.933463 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260812 11:20:40.933751 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.921070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.27.109.61:32789)
0812 11:20:40.921297 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.921302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.921322 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:20:40.921412 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.921416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.921487 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.921642 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.922391 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.923382 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.924633 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.925014 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.925170 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.928633 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.928640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.928644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.928754 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.931073 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.931770 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.931775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.931778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.931891 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.932480 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.932484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.932486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.932777 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.932864 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.933388 (+ 524us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.933556 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 11:20:40.934140 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.921661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.27.109.62:45665)
0812 11:20:40.921818 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.921822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.921833 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:20:40.921877 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.921879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.921923 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.922004 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.922891 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.923792 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.924920 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.925357 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.925529 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.929382 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.929386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.929389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.929428 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.931324 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.932076 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.932080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.932082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.932207 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.932663 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.932666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.932668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.933053 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.933375 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.933746 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.933927 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:20:40.935195 32412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260812 11:20:40.948601 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.936060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60678)
0812 11:20:40.936298 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.936315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.936463 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.936840 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.936844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.937080 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.937206 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.937213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.939229 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.939236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.940253 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.940268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.941029 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.941033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.941130 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.941826 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.941848 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.943994 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.946345 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.946350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.946358 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.946638 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.947073 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.947076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.947078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.947223 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.947838 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.947841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.947942 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.948064 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.948316 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260812 11:20:40.948606 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.936164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.27.109.60:46673)
0812 11:20:40.936301 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.936307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.936382 (+ 75us) server_negotiation.cc:407] Connection header received
0812 11:20:40.936624 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.936631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.936711 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.936820 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.937349 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.939096 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.940453 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.940804 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.940998 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.944283 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.944290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.944293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.944334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.946182 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.946793 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.946797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.946800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.946940 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.947360 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.947364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.947367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.947713 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.947856 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.948137 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.948387 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":69}
I20260812 11:20:40.949292 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.936857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.27.109.61:32789)
0812 11:20:40.936984 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.936987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.937648 (+ 661us) server_negotiation.cc:407] Connection header received
0812 11:20:40.937833 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.937836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.937901 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.938022 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.939137 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.939727 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.940900 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.941216 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.941351 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.944997 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.945005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.945010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.945061 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.947214 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.947770 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.947773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.947774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.947861 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.948165 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.948168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.948170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.948393 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.948700 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.948795 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.948921 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:20:40.954816 32412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: Bootstrap starting.
I20260812 11:20:40.955476 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:20:40.957369 32150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52344
I20260812 11:20:40.958712 32412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:40.959829 32412 log.cc:824] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:40.963430 32412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: No bootstrap required, opened a new log
I20260812 11:20:40.967834 32412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
I20260812 11:20:40.968250 32412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8672854f9844579a02a635edd567e9, State: Initialized, Role: FOLLOWER
I20260812 11:20:40.968711 32412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
I20260812 11:20:40.969610 32420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:40.969898 32420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:40.971061 32412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:40.978359 32198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } attempt: 3
W20260812 11:20:40.979660 32436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:40.979776 32436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:40.977427 32308 master_runner.cc:186] Error getting master registration for 127.27.109.62:45665: OK, instance_id { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" instance_seqno: 1786533640315682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:20:40.983706 32308 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 11:20:40 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260812 11:20:40.991554 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.979125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60684)
0812 11:20:40.979348 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:40.979367 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:40.979503 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:40.979797 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:40.979801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:40.980069 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:40.980201 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.980208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.981316 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.981320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.981997 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:40.982009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.982657 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.982660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:40.982748 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.983525 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:40.983542 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:40.985122 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:40.988106 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.988113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.988115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.988622 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.989108 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:40.989112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:40.989114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:40.989248 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:40.990174 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:40.990181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:40.990587 (+ 406us) client_negotiation.cc:769] Sending connection context
0812 11:20:40.990791 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 11:20:40.991068 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260812 11:20:40.991647 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:40.979084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.27.109.60:46673)
0812 11:20:40.979245 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:40.979250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:40.979441 (+ 191us) server_negotiation.cc:407] Connection header received
0812 11:20:40.979592 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:40.979596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:40.979651 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:40.979766 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:40.980357 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.981195 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.982172 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:40.982529 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:40.982644 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:40.985365 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:40.985372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:40.985376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:40.985430 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:40.987834 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.988812 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.988817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.988820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.988959 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:40.989416 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:40.989422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:40.989424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:40.989884 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:40.990039 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:40.991078 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0812 11:20:40.991325 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:20:40.999183 32198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0: Bootstrap starting.
I20260812 11:20:41.004782 32198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:41.006531 32198 log.cc:824] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:20:41.007831 32300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } attempt: 4
I20260812 11:20:41.009968 32198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0: No bootstrap required, opened a new log
I20260812 11:20:41.014245 32198 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
I20260812 11:20:41.014619 32198 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b5d793d9fd4f46a7ce14c64f6d7be0, State: Initialized, Role: FOLLOWER
I20260812 11:20:41.015093 32198 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
Aug 12 11:20:41 dist-test-slave-7z32 krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533640, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260812 11:20:41.016834 32439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.017176 32439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:41.017405 32198 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:41.020646 32150 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 11:20:41.021260 32150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:41.024135 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:41.008825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.27.109.60:46673)
0812 11:20:41.008995 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.009001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.009035 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:20:41.009211 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.009215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.009281 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.009423 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.010069 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.010986 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.012480 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.012932 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.013097 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:41.016911 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:41.016922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:41.016928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:41.017046 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:41.020350 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.021565 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.021572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.021574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.021743 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.022358 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.022363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.022364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.022716 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:41.022965 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:41.023396 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 11:20:41.023576 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:20:41.024505 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:41.008636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46673 (local address 127.0.0.1:60694)
0812 11:20:41.008928 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 11:20:41.008950 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:20:41.009065 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:20:41.009473 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:20:41.009479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:20:41.009735 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:20:41.009885 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:41.009894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.011272 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.011278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:41.012227 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:20:41.012244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.013136 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.013140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:20:41.013252 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:41.014127 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:20:41.014166 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:20:41.016501 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:20:41.020639 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:41.020646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:41.020650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:41.021262 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:41.021905 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:20:41.021908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:20:41.021911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:20:41.022093 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:20:41.022966 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:20:41.022970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:20:41.023150 (+ 180us) client_negotiation.cc:769] Sending connection context
0812 11:20:41.023289 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 11:20:41.024257 (+ 968us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260812 11:20:41.024073 32150 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
I20260812 11:20:41.029605 32150 leader_election.cc:288] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [CANDIDATE]: Term 1 election: Requested vote from peers afda5da8847d4d308733a6288b19b0c4 (127.27.109.61:32789), 4c8672854f9844579a02a635edd567e9 (127.27.109.60:46673)
I20260812 11:20:41.032891 32300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: Bootstrap starting.
I20260812 11:20:41.035701 32376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c8672854f9844579a02a635edd567e9"
I20260812 11:20:41.036181 32376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:41.038313 32376 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b5d793d9fd4f46a7ce14c64f6d7be0 in term 1.
I20260812 11:20:41.039117 32121 leader_election.cc:302] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8672854f9844579a02a635edd567e9, f8b5d793d9fd4f46a7ce14c64f6d7be0; no voters:
I20260812 11:20:41.039618 32439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:20:41.040026 32439 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [term 1 LEADER]: Becoming Leader. State: Replica: f8b5d793d9fd4f46a7ce14c64f6d7be0, State: Running, Role: LEADER
I20260812 11:20:41.035722 32094 master_runner.cc:186] Error getting master registration for 127.27.109.61:32789: OK, instance_id { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" instance_seqno: 1786533640585002 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:20:41.040491 32439 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
I20260812 11:20:41.043308 32439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b5d793d9fd4f46a7ce14c64f6d7be0. Latest consensus state: current_term: 1 leader_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.043620 32439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:41.043836 32300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: Neither blocks nor log segments found. Creating new log.
I20260812 11:20:41.040493 32094 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:20:41.045383 32300 log.cc:824] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: Log is configured to *not* fsync() on all Append() calls
W20260812 11:20:41.046167 32452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:41.046268 32452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:41.046550 32463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:20:41.048096 32463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:20:41.049019 32300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: No bootstrap required, opened a new log
I20260812 11:20:41.054175 32300 raft_consensus.cc:348] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
I20260812 11:20:41.054713 32300 raft_consensus.cc:729] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afda5da8847d4d308733a6288b19b0c4, State: Initialized, Role: FOLLOWER
I20260812 11:20:41.055518 32300 consensus_queue.cc:260] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } }
I20260812 11:20:41.057886 32300 sys_catalog.cc:564] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:20:41.057996 32268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "afda5da8847d4d308733a6288b19b0c4"
I20260812 11:20:41.058291 32268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:20:41.060153 32268 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b5d793d9fd4f46a7ce14c64f6d7be0 in term 1.
W20260812 11:20:41.060802 32125 tablet.cc:2369] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:20:41.067081 32487 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:20:41.067289 32487 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:20:41.068327 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:41.048194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.27.109.60:46673)
0812 11:20:41.048330 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.048334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.048978 (+ 644us) server_negotiation.cc:407] Connection header received
0812 11:20:41.050418 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.050445 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.050527 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.050607 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.052098 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.052922 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.055755 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.056611 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.057053 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:41.060056 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:41.060068 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:41.060071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:41.060110 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:41.062047 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.064387 (+ 2340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.064392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.064394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.064752 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.065947 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.065953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.065958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.066291 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:41.066462 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:41.067752 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0812 11:20:41.067964 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:20:41.072019 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:41.047176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.27.109.62:45665)
0812 11:20:41.047306 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.047311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.048500 (+ 1189us) server_negotiation.cc:407] Connection header received
0812 11:20:41.048760 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.048764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.048826 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.048974 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.050244 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.051072 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.053497 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.053830 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.053923 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:41.061840 (+ 7917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:41.061846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:41.061852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:41.061884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:41.063782 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.065376 (+ 1594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.065383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.065388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.065539 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.066886 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.066891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.066892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.067178 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:41.067323 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:41.071487 (+ 4164us) server_negotiation.cc:299] Negotiation successful
0812 11:20:41.071775 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:20:41.078141 32376 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [term 1 FOLLOWER]: Refusing update from remote peer f8b5d793d9fd4f46a7ce14c64f6d7be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:20:41.079730 32268 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [term 1 FOLLOWER]: Refusing update from remote peer f8b5d793d9fd4f46a7ce14c64f6d7be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 11:20:41.080911 32242 tablet.cc:2369] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:20:41.081282 32458 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:20:41.087361 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.087591 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:41.087807 32458 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:20:41.088564 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:41.051387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.27.109.61:32789)
0812 11:20:41.051585 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.051592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.060241 (+ 8649us) server_negotiation.cc:407] Connection header received
0812 11:20:41.060472 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.060476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.060539 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.060608 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.061745 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.062567 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.064034 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:41.064413 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:41.064692 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:41.083073 (+ 18381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:20:41.083082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:20:41.083086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:20:41.083158 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:20:41.085496 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.086361 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.086366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.086368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.086488 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:20:41.086962 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:20:41.086966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:20:41.086968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:20:41.087286 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:20:41.087589 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:41.088173 (+ 584us) server_negotiation.cc:299] Negotiation successful
0812 11:20:41.088371 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:20:41.097822 32420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b5d793d9fd4f46a7ce14c64f6d7be0. Latest consensus state: current_term: 1 leader_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.098202 32420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:41.101192 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b5d793d9fd4f46a7ce14c64f6d7be0. Latest consensus state: current_term: 1 leader_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.101527 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:41.120394 32458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.120687 32458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:41.121526 32420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.121721 32420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:41.126421 32491 mvcc.cc:205] Tried to move back new op lower bound from 7317641793845186560 to 7317641793706778624. Current Snapshot: MvccSnapshot[applied={T|T < 7317641793845186560}]
I20260812 11:20:41.127161 32439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.127380 32439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:20:41.129226 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5d793d9fd4f46a7ce14c64f6d7be0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45665 } } peers { permanent_uuid: "afda5da8847d4d308733a6288b19b0c4" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32789 } } peers { permanent_uuid: "4c8672854f9844579a02a635edd567e9" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46673 } } }
I20260812 11:20:41.129428 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:20:41.134382 32490 mvcc.cc:205] Tried to move back new op lower bound from 7317641793845186560 to 7317641793706778624. Current Snapshot: MvccSnapshot[applied={T|T < 7317641793845186560}]
I20260812 11:20:41.134791 32463 catalog_manager.cc:1347] Generated new cluster ID: c8bbba235c1e4bd29c9b17d5ece8cc8e
I20260812 11:20:41.134897 32463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:20:41.169966 32463 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:20:41.171906 32463 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:20:41.214967 32463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b5d793d9fd4f46a7ce14c64f6d7be0: Generated new TSK 0
I20260812 11:20:41.216491 32463 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:20:41.258919 32203 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.253812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.27.109.62:45665)
0812 11:20:41.253994 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.254000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.257178 (+ 3178us) server_negotiation.cc:407] Connection header received
0812 11:20:41.257525 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.257530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.257608 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.257677 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.258683 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52362: BlockingRecv error: recv got EOF from 127.0.0.1:52362 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 11:20:41.262157 32416 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.255615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.27.109.60:46673)
0812 11:20:41.256040 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.256046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.260473 (+ 4427us) server_negotiation.cc:407] Connection header received
0812 11:20:41.260941 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.260946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.261023 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.261101 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.261942 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60712: BlockingRecv error: recv got EOF from 127.0.0.1:60712 (error 108)
Metrics: {"server-negotiator.queue_time_us":301}
W20260812 11:20:41.264329 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.259810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.27.109.61:32789)
0812 11:20:41.259968 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.259974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.263073 (+ 3099us) server_negotiation.cc:407] Connection header received
0812 11:20:41.263308 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.263314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.263384 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.263447 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.264143 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57770: BlockingRecv error: recv got EOF from 127.0.0.1:57770 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 11:20:41.271757 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.269503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.27.109.61:32789)
0812 11:20:41.269696 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.269702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.270242 (+ 540us) server_negotiation.cc:407] Connection header received
0812 11:20:41.270573 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.270577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.270700 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.270769 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.271576 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57778: BlockingRecv error: recv got EOF from 127.0.0.1:57778 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260812 11:20:41.279728 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.277071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.27.109.61:32789)
0812 11:20:41.277265 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.277275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.278199 (+ 924us) server_negotiation.cc:407] Connection header received
0812 11:20:41.278606 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.278611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.278683 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.278749 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.279558 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57786: BlockingRecv error: recv got EOF from 127.0.0.1:57786 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 11:20:41.288079 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.285660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.27.109.61:32789)
0812 11:20:41.285844 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.285851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.286665 (+ 814us) server_negotiation.cc:407] Connection header received
0812 11:20:41.286992 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.286997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.287074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.287141 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.287911 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57798: BlockingRecv error: recv got EOF from 127.0.0.1:57798 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 11:20:41.297914 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.296025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.27.109.61:32789)
0812 11:20:41.296195 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.296200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.296651 (+ 451us) server_negotiation.cc:407] Connection header received
0812 11:20:41.296839 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.296843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.296933 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.297055 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.297766 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57810: BlockingRecv error: recv got EOF from 127.0.0.1:57810 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 11:20:41.310721 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.309037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.27.109.61:32789)
0812 11:20:41.309221 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.309227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.309491 (+ 264us) server_negotiation.cc:407] Connection header received
0812 11:20:41.309666 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.309670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.309752 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.309905 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.310549 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57816: BlockingRecv error: recv got EOF from 127.0.0.1:57816 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260812 11:20:41.332392 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.329918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.27.109.61:32789)
0812 11:20:41.330095 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.330101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.330498 (+ 397us) server_negotiation.cc:407] Connection header received
0812 11:20:41.330881 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.330887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.330969 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.331120 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.332221 (+ 1101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57822: BlockingRecv error: recv got EOF from 127.0.0.1:57822 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 11:20:41.369117 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.367283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.27.109.61:32789)
0812 11:20:41.367658 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.367664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.367703 (+ 39us) server_negotiation.cc:407] Connection header received
0812 11:20:41.367920 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.367923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.367989 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.368135 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.368913 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57834: BlockingRecv error: recv got EOF from 127.0.0.1:57834 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260812 11:20:41.437866 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.435991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.27.109.61:32789)
0812 11:20:41.436205 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.436210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.436553 (+ 343us) server_negotiation.cc:407] Connection header received
0812 11:20:41.436639 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.436641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.436703 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.436811 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.437700 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57850: BlockingRecv error: recv got EOF from 127.0.0.1:57850 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260812 11:20:41.571261 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.569322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.27.109.61:32789)
0812 11:20:41.569673 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.569678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.569997 (+ 319us) server_negotiation.cc:407] Connection header received
0812 11:20:41.570166 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.570169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.570232 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.570326 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.571110 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57862: BlockingRecv error: recv got EOF from 127.0.0.1:57862 (error 108)
Metrics: {"server-negotiator.queue_time_us":224}
W20260812 11:20:41.832217 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:20:41.830546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.27.109.61:32789)
0812 11:20:41.830702 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:41.830706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:41.830854 (+ 148us) server_negotiation.cc:407] Connection header received
0812 11:20:41.831142 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:41.831145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:41.831206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:41.831321 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:20:41.832079 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57864: BlockingRecv error: recv got EOF from 127.0.0.1:57864 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:20:41.981376 32436 catalog_manager.cc:1259] Loaded cluster ID: c8bbba235c1e4bd29c9b17d5ece8cc8e
I20260812 11:20:41.981487 32436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: loading cluster ID for follower catalog manager: success
I20260812 11:20:41.983402 32436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: acquiring CA information for follower catalog manager: success
I20260812 11:20:41.984735 32436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c8672854f9844579a02a635edd567e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:20:42.068938 32487 catalog_manager.cc:1259] Loaded cluster ID: c8bbba235c1e4bd29c9b17d5ece8cc8e
I20260812 11:20:42.069049 32487 catalog_manager.cc:1552] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: loading cluster ID for follower catalog manager: success
I20260812 11:20:42.070765 32487 catalog_manager.cc:1574] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: acquiring CA information for follower catalog manager: success
I20260812 11:20:42.072230 32487 catalog_manager.cc:1602] T 00000000000000000000000000000000 P afda5da8847d4d308733a6288b19b0c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:20:42.352648 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:42.347531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.27.109.61:32789)
0812 11:20:42.347927 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:42.347932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:42.347959 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:20:42.348038 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:42.348041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:42.348242 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:42.348407 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:42.348785 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.349439 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.350482 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.350750 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.350971 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:42.351424 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:42.352056 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:42.352175 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:42.352316 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 11:20:42.352391 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260812 11:20:42.362030 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:42.356580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.27.109.60:46673)
0812 11:20:42.356935 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:42.356940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:42.356957 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:20:42.357189 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:42.357192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:42.357359 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:42.357491 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:42.357737 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.358410 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.359744 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.360040 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.360149 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:42.360667 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:42.361348 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:42.361501 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:42.361639 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 11:20:42.361791 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":129,"threads_started":1}
I20260812 11:20:42.371186 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:42.366862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.27.109.61:32789)
0812 11:20:42.367039 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:42.367043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:42.367215 (+ 172us) server_negotiation.cc:407] Connection header received
0812 11:20:42.367478 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:42.367481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:42.367544 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:42.367644 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:42.367934 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.368724 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.369835 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.370096 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.370226 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:42.370576 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:42.370664 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:42.370757 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:42.370924 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 11:20:42.370991 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:20:42.372817 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:42.366575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.27.109.60:46673)
0812 11:20:42.366733 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:42.366743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:42.367443 (+ 700us) server_negotiation.cc:407] Connection header received
0812 11:20:42.367639 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:42.367643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:42.367708 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:42.367758 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:42.368237 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.369094 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.370951 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.371344 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.371442 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:42.372119 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:42.372239 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:42.372358 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:42.372534 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 11:20:42.372644 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:20:42.373596 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:20:42.366526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.27.109.62:45665)
0812 11:20:42.367211 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 11:20:42.367217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:20:42.367573 (+ 356us) server_negotiation.cc:407] Connection header received
0812 11:20:42.367735 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:20:42.367742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:20:42.367974 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:20:42.368148 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:20:42.368456 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.369261 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.370949 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:20:42.371369 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:20:42.371463 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:20:42.372200 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:20:42.372800 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:20:42.372914 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:20:42.373297 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 11:20:42.373403 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":103,"threads_started":1}
I20260812 11:20:42.388481 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 32094
I20260812 11:20:42.416520 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 32211
I20260812 11:20:42.431363 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9g2P8/build/debug/bin/kudu with pid 32308
2026-08-12T11:20:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3951 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7805 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (359620 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST